Solved: OpsGenie alerts to different Slack channels based on priority

When setting up the excellent OpsGenie to do the daily job of gathering alerts from various sources and routing and escalating the alerts based on the rules you set to your team, one option is to forward alerts to Slack via a Slack integration. This is very convenient if you have implemented Slack culture in your organization and people are using Slack on their desktops and mobile phones too.

It’s a great setup but after a while you might notice something happening, which is alert/alarm fatigue. If all different priority alerts are pushed to your one Slack channel, then after a while people responsible for reading the alerts will be desensitized and thus may not react in a proper way once an alert is present in the alerts channel.

How to prevent alert fatigue in an alerts channel in Slack?

Workaround is to separate the critical and lower priority alerts to dirrefent Slack channels. I did so and set up channels #alerts-p1, #alerts-p2 etc for each level of alerts. This way people responsible for reading the alerts can set their notification settings per channel.

How to send OpsGenie alerts to different Slack channels?

In OpsGenie when configuring an integration (i.e. Amazon AWS, GhostInspector, New Relic etc) you can set up priorities for the alerts coming from those services.

Alerts are on OG then prioritized based on your rules like:

  • P1-Critical
  • P2-High
  • P3-Moderate
  • etc

Now when you are configuring the Slack integration to send the alerts to a Slack channel you are able to select only one Slack channel per your integration. A shame.

But be clever! To make alerts go to another Slack channel, just set up a new Slack integration where you can set the teams and the Slack channel that the alerts will go.

Also – please note – you then apply (at the bottom of the integration) a FILTER to the integration. Select “one or more conditions match” and set the condition (priority) and the corresponding priority for your Slack channel / Slack integration.

Then go to OpsGenie -> Alerts and create a new alert (for testing). Set different priorities to test alerts to see how they flow in your Slack to different channels.

That was easy, wasn’t it!

Post Author: Tom Pai