Question
IBM Cloud Monitoring Duplicate Notifications from Overlapping Alert Policies
Nia Sage
0 reputation · 27 Feb 2023, 02:26 UTC
56.9K views1
Loading…
0 reputation · 27 Feb 2023, 02:26 UTC
When configuring IBM Cloud Monitoring alerts using Sysdig's alerting engine, teams often create multiple notification policies to route alerts to different channels based on severity, scope, or custom labels. However, a documented behavior exists where each policy evaluates alerts independently against its matching criteria.
If multiple policies have overlapping label selectors or severity thresholds, a single alert can match more than one policy. This causes duplicate notifications to be sent to the same channel when the alert fires. For example, an alert tagged with team=payments and severity=critical might match both a 'critical-severity' policy and a 'payments-team' policy, resulting in the same Slack channel receiving two identical messages.
The grouping mechanism in IBM Cloud Monitoring allows collapsing multiple alerts into a single notification using 'group by' fields and a configurable time window, but this grouping occurs within each policy independently. When policies overlap, the same alert fires multiple times across policies, bypassing any intended deduplication at the policy level.
This behavior creates notification noise that undermines the purpose of alert grouping and can lead to alert fatigue for on-call engineers. The documentation does not specify a policy evaluation order or provide a mechanism to ensure mutual exclusivity between policies.
34775 reputation · 27 Feb 2023, 08:10 UTC
Is there a documented method to enforce mutual exclusivity? No. The Sysdig-derived alerting engine evaluates each notification policy independently against its own matching criteria. There is no first-match-wins, priority, or stop-processing toggle in the current IBM Cloud Monitoring offering.
Does policy evaluation order matter? No. Matching is set-based, not sequential. Order is not documented as deterministic and cannot be relied upon to suppress duplicates.
How to design non-overlapping routing? Make policy matching criteria pairwise disjoint for any policies that share a destination channel. Use a single routing dimension per policy set (team label or severity, not both), or add explicit exclusion labels so each alert satisfies exactly one policy's selector.
Duplicates that appear "identical" may also come from the same alert re-notifying on repeat/resolved transitions within one policy. Before redesigning policies, trigger a test alert that matches two policies and compare the policy name/ID fields in both received messages.
ibmteam=payments; Policy 2 matches team=platform. Do not also match on severity in both.severity!=critical) so an alert cannot satisfy both.If you must tolerate overlapping policies, dedupe at the receiver using the alert's unique ID/fingerprint present in the webhook payload. This is a workaround, not a platform feature.
Do the duplicate messages carry different policy names/IDs in the payload, or the same one? This determines whether the fix is policy redesign (different policies) or repeat-notification tuning (same policy).
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.