IBM Cloud Monitoring Duplicate Notifications from Overlapping Alert Policies

Problem

When configuring IBM Cloud Monitoring alerts using Sysdig's alerting engine, teams often create multiple notification policies to route alerts to different channels based on severity, scope, or custom labels. However, a documented behavior exists where each policy evaluates alerts independently against its matching criteria.

If multiple policies have overlapping label selectors or severity thresholds, a single alert can match more than one policy. This causes duplicate notifications to be sent to the same channel when the alert fires. For example, an alert tagged with team=payments and severity=critical might match both a 'critical-severity' policy and a 'payments-team' policy, resulting in the same Slack channel receiving two identical messages.

The grouping mechanism in IBM Cloud Monitoring allows collapsing multiple alerts into a single notification using 'group by' fields and a configurable time window, but this grouping occurs within each policy independently. When policies overlap, the same alert fires multiple times across policies, bypassing any intended deduplication at the policy level.

This behavior creates notification noise that undermines the purpose of alert grouping and can lead to alert fatigue for on-call engineers. The documentation does not specify a policy evaluation order or provide a mechanism to ensure mutual exclusivity between policies.

Key Questions

• Is there a documented method to enforce mutual exclusivity between notification policies to prevent duplicate deliveries?
• Does the policy evaluation order matter for alert routing, and is it deterministic?
• How can teams design their notification policies to avoid overlap while maintaining flexible routing rules?