Alert notifications flow

This doc will bring you through the flow of an Alert once created, based on the graphic below. Follow along with the graphic's labeled pipelines (1-18) to understand the typical alert flow and the many ways Opsgenie allows you to configure who get's notified, and when.

Opsgenie’s Alert features help you build and optimize your service-aware incident management and management systems and processes. Integrate Opsgenie Alerts with monitoring systems, service management and ticketing solutions, collaboration tools, and other teaming tools. Opsgenie filters, categorizes, and enriches Alert content —  in addition to combining related Alerts (into Incidents) and adding attachments, notes, and links —  in order to maximize available information about incidents and minimize the time and trouble needed to evaluate Alerts and decide what to do about them. Easily collaborate on Alerts and take action on them in a variety of channels, knowing that Opsgenie synchronizes actions or performs custom actions that are easily mapped.

Opsgenie provides even more ways to see what is happening in a system —  and to optimize an organization’s ability to quickly respond to events. Customizing Opsgenie Escalations, On-Call Scheduling, Rules, and Policies can enhance incident response mechanisms and processes. Understanding the Alert Notifications flow can help users see the opportunities to continue refining incident response processes. The diagram and descriptions below show an overview of this flow, and the descriptions of each step contain links to more information about the process associated with that step.

1. Alert is Created

The Alert Creation Flow flow precedes the Alert Notifications flow and determines whether —  and when —  the Alert Notifications flow begins. When the Alert Creation flow initiates the Alert Notifications flow, the Status of the Alert is Open.

a. Alert Creation Flow
During the Alert Creation flow, Opsgenie processes rules, policies, and other operations on incoming data from all sources (incoming Alerts) before the Alert Notifications flow begins.

For example:

i. Filtering and Suppressing Alerts
Opsgenie Integrations can filter incoming alerts and create Alerts or perform other actions (including ignoring incoming alerts) based on matches to configurable conditions. Other Integration rules and settings can suppress Alerts before the Notifications flow begins: no Notifications are sent, but users can access the Alert.

ii. Notification Policies
Notification Policies can delay the beginning of the Alert Notifications flow. Opsgenie checks incoming alerts from all sources during the Alert Creation flow and can be configured to delay the Alert Notifications flow until a source has attempted to create an Opsgenie Alert with the same Alias (unique identifier for Alerts) a set number of times.

iii. Deduplication
If incoming alerts from any source attempt to create a new Opsgenie Alert when there is already an open Alert with the same Alias, Opsgenie deduplicates the Alert. Deduplication increases an Alert’s Alert Count value by one, instead of creating another Alert. Opsgenie adds these events to the Alert’s Activity Log (to enhance tracking).

Although Opsgenie continues to deduplicate Alerts (as long as their state is Open), the Alert Count value and Activity Log stop showing deduplication activity after 100 occurrences.

b. Alert State
When an Alert is first created —  manually or through an integrated tool —  the Alert’s state is Open and Unacknowledged. Over the course of an Alert’s lifecycle, its state changes as a result of the operations of rules, policies, and user actions.

Whether Opsgenie sends an Alert Notification depends on the state of an Alert.

Opsgenie checks an Alert’s state during every step of the Alert Notifications flow. The Notification flow of any Alert ends if Opsgenie determines that an Alert is Closed —  through actions of a user, an Auto-Close Policy, or an Integration Close Alert Rule.

2. Auto-Close Policies Processed

When Alert content matches conditions of an Auto-Close Policy, an Auto-Close event is triggered. When the time set for an Auto-Close event arrives, Opsgenie automatically closes the Alert.

3. Auto Restart Notification Policies Processed

If you set an Auto Restart Notification Policy, the current Notification flow is discarded and restarted (regardless of the current Recipient or Escalation states) following a policy-specified time after an Alert’s creation —  as long as the alert is not Closed. So, an Open Alert moves on to Step 3, regardless of its current state.

The following policies are also re-triggered and their time states are refreshed:

• Auto Close

• Notification Policies with Delay or Suppress options

4. - 8. Who Should be Notified?

What happens to Alerts next depends on whether immediate Alert Recipients or Teams are identified in the Alert.

• If only immediate/individual Recipients or both individual Recipients and Teams are specified in Alert content (4), they are added to the Alert Notification (8).

• If only Teams (no immediate Recipients) are specified in Alert content (5), then what happens next depends on whether that team has a matching Routing Rule (6).

  • If there is no matching Routing Rule, then no Notification is sent (Team members can still view and access the Alert).

  • If there is a matching Routing Rule, then the first matching Routing Rule routes the Alert Notification according to the Team’s Escalation or On-Call Schedule -- or to no one: Team members can still view and access the Alert (but no Notification is sent). (7)

Routing Rules are not triggered if even one individual Recipient is specified in an Alert. The Alert Notifications flow moves on to (11) (if an organization’s Opsgenie plan includes Policies) or to (15) (if the plan does not include Policies).

9. - 10. Escalations Processed

If an Escalation is triggered (no matter when the Escalation was added to the Recipients field), Opsgenie starts processing Escalation Rules (9) to see whether to add new Recipients to an Alert Notification. Opsgenie checks these rules against Alert states.

For example:

• If an Escalation Rule notifies Recipients only about Alerts that have not been Acknowledged (or that have been Unacknowledged), Opsgenie only adds new Recipients to an Alert Notification if the Alert is both Open and Unacknowledged (“if not acked”).

• If the Escalation Rule notifies Recipients only when the Alert is not Closed, target Recipients are added to the Alert if the alert is Open -- even if the Alert has been Acknowledged.

If an Escalation Rule adds any Teams as new Recipients, the Notifications flow starts for the added Team Recipients at Step 6. If it adds any Individual Recipients, the Notifications flow begins for these new Recipients at Step 8 —  whether these Recipients are single users, other members of a team, or members of an on-call schedule.

11. - 14. Notification Policies Processed

Some Opsgenie plans let an organization’s Administrative users apply different operations to Alert Notifications by means of Notification Policies. If an organization’s plan does not include Notification Policies, then the Notifications flow continues at (15). If an Alert’s content matches a Notification Policy’s conditions, however, Opsgenie processes Notification Policies (11).

Users set Notification Rules (related to how Alert Notifications are delivered to them) on the Notifications tabs of their own Profiles. Opsgenie processes Recipient Notifications settings in (17).

Were Notifications Suppressed (12)?
If a Notification Policy suppresses an Alert Notification, Opsgenie does not notify any of the Alert’s Recipients, although users can still access and take action on suppressed Alerts.

Integrations can suppress Alerts during the Alert Creation flow. The Alerts Notifications flow does not begin when an Integration suppresses an Alert during the Alert Creation flow.

Were Notifications Delayed (13)?
If a matching Notification Policy delays Notifications, a delay event is triggered. When the time set for the delay ends (14), the flow continues.

Notification Policies can also delay Alerts during the Alert Creation flow. When Alerts are delayed during the Alert Creation flow, then an Alert is not created —  and the Alert Notifications flow does not begin —  until a source has repeatedly attempted to create an Opsgenie Alert with the same Alias (a configurable number of times).

15. Alert State and Recipient Readiness are checked

Before Opsgenie sends Notifications, it checks to make sure that Recipients are ready.

A Recipient is ready:

• Whenever the Recipient was added -- as long as no matching Policy adds a delay.

• Whenever the Recipient was added plus any delay time -- if a matching Policy adds a delay.

Opsgenie then checks the state of Alerts, one more time:

16. User Notification Settings Processed

On the Notifications tabs of their Profiles, users can set Notification Rules that determine how Opsgenie delivers Alert Notifications to them. Opsgenie processes these user Notification Settings according to the following priorities:

If a Recipient has set Forwarding Rules, Opsgenie forwards the Alert Notification to the new Recipient for a set time frame: the Notifications flow ends for the original, replacing the previous Recipient and adding the substituted Recipient in the flow before Step 15. Even after the period for the forwarding rule expires, the previous Recipient will no longer receive updates for that Alert Notification.

For the Add Note, Acknowledge and Close actions, Opsgenie sends a notification only if recipient is aware of the alert. This means, either recipients has already been notified about the alert, or they have seen the alert.

17. Is a Contact Method Enabled for a Matching Notification Rule?

Opsgenie can only send an Alert Notification if a user has enabled a Contact Method for a matching Notification Rule on the Notification Settings tab in their Profile. If there is no matching Notification Rule or if no Contact Method is enabled for the matching Notification Rule, then Opsgenie will not send an Alert.

18. Alert Notification Sent

Opsgenie sends the Alert Notification to one or more Recipients according to the Contact Method selected by each user.