Alerts lets you create customized alerting solutions for monitoring your system.
When you set up alerts, you can get notified about problems you're interested in.
To help you get started, here are some alerts concepts and the workflow you'll follow.
Introduction to important concepts
To use alerts well, it will help you to understand the general flow of how the conditions and policies you create lead to violations and notifications.
To use alerts well, it will help you to understand the terms we use:
Alerts terminology | Comments |
|---|---|
Policy | A policy is a group of one or more alert conditions. You must create a policy before you can add conditions to it. A policy has two settings that apply to all of its conditions: incident preference and notification channels (explained more below). |
Condition | A condition includes: a) a monitored data source and b) thresholds that define the behavior that's considered a violation. For example, a specific condition might be described in this way: "If the response time for any page load in my app goes above 8 seconds and lasts for more than 5 minutes, that's a violation." |
Threshold | A threshold is part of a condition; it defines the behavior that's considered a violation. When you create a condition, there's a required critical-level threshold. Optionally, you can set a secondary warning-level threshold. |
Violation | A violation occurs when the value of a data source crosses a condition's threshold. This leads to the creation of a violation event, which is used to pass important information downstream. A violation doesn't directly generate a notification; a violation may lead to an incident, which in turn can generate notifications. |
Incident | Incidents are what generate notifications. At the policy level, the incident preference determines how violations are handled and combined to generate incidents. For example, you may want to have every single violation generate an incident (many notifications) or you may want to have only a single incident open at a time across an entire alert policy (minimal notifications). Setting the incident preference gives you power over how notifications are created and helps prevent notification fatigue. |
Notification | At the policy level, you choose what team members are notified when an incident occurs and how they're notified. We offer several notification channels, including webhooks, Slack rooms, email, etc. You can include charts about the incident to provide context, and share them with your team's notification. |
For in-depth definitions of these and other terms, see the glossary.
Basic workflow
Now that you understand some basic concepts and terms, let's look at a typical process for creating a policy and an associated condition:
Create a policy. When you create a policy:
- Give it a meaningful name. For example: the group or team's name, or the set of resources or services the policy targets.
- Set the incident preference, which determines how violations become incidents.
- Set notification channels.
Create a condition that will be attached to that policy. Steps involved in creating a condition include:
- Choose a data source that will be monitored (for example, an APM metric or a NRQL query).
- Set the thresholds that define what behavior will produce a violation.
- Optional: Include a runbook URL, which is used to share standard procedures for how to handle alert notifications.
Optional: Add more conditions to that same policy.
In addition to receiving notifications, you can view the alert incident or event details in the UI.
What's next?
To learn more about using alerts:
- Read our best practices.
- Learn about the API.