Overlapping silence rules delaying grouped notifications in Grafana Alertmanager

Direct Answer

An overlapping silence suppresses only the alerts whose labels match the silence matchers, not the entire notification group. However, because Alertmanager groups alerts before evaluating silences, a silenced alert can still create or join an aggregation group and start the group_wait timer. Unsilenced critical alerts that share the same group key are then delayed until that timer expires. This assumes a Prometheus-compatible Alertmanager as used by Grafana Alerting; timer semantics can vary slightly across versions, so confirm behavior in your specific deployment.

What Is Confirmed vs. Likely

Confirmed behavior in a Prometheus-compatible Alertmanager: silences are applied per alert after grouping. The silence removes matching alerts from the payload, but it does not prevent those alerts from participating in the group aggregation or from triggering the group schedule.

The likely explanation for your delay is that a silenced maintenance alert arrived first, created the group, and initiated group_wait. The critical alert joined that same group and was buffered until the group flushed.

Effect of Group Wait and Group Interval

• Group Wait buffers the first notification for the whole aggregation group. If a silenced alert arrives first and shares the group key with a critical alert, the critical alert is delayed until group_wait expires, even though the silenced alert is omitted from the payload.
• Group Interval throttles subsequent notifications for an existing group. If a critical alert is added to a group that is already within its group_interval or repeat_interval window, delivery may be postponed until the next scheduled send rather than sent immediately.

Configuration Pattern to Prevent Delays

1. Route critical alerts separately using distinct group_by labels—for example, adding severity or alertname—so they do not share a group with broad maintenance alerts.
2. Configure a shorter group_wait on that critical route (e.g., 0s or 10s).
3. Scope silence matchers narrowly to maintenance-specific labels instead of common group labels.
4. Consider inhibition rules instead of silences for maintenance scenarios; they are also evaluated per alert and can target source alerts without affecting unrelated alerts that share group labels.

Verification

To confirm the behavior in your deployment, create a test route with group_by: [alertname] and fire two alerts sharing that label. Create a silence matching only one, then observe whether the unsilenced alert is delayed by the configured group_wait. Use the Alertmanager API or Grafana Alerting UI to inspect active silences and verify which alerts are matched, then compare notification timestamps against alert firing times. Review Alertmanager logs or metrics for group flush events to confirm that the group timer started when the silenced alert arrived.

Missing Diagnostic

What are the current group_by labels and group_wait value for the affected route? If group_wait is already 0s, the delay is likely caused by group_interval or repeat_interval rather than the initial buffer, which changes the recommended fix.