Question
Prometheus HA Replicas and Alertmanager Deduplication: Label-Set Identity Across Replicas
Nia Maple
0 reputation · 28 Feb 2021, 04:52 UTC
69.2K views1
Loading…
0 reputation · 28 Feb 2021, 04:52 UTC
Prometheus HA replicas each expose firing and resolved alerts to a shared Alertmanager via the Prometheus Alerting API at /api/v1/alerts. Alertmanager deduplicates incoming alerts by comparing label sets, treating alerts with identical labels as a single group for notification purposes.
The deduplication logic is version-sensitive and depends on whether replicas share identical external_labels. When replicas carry divergent external_labels, the same alerting rule produces distinct alert identities in Alertmanager, potentially triggering duplicate notifications. Annotations are excluded from the grouping key, yet alerts differing only in annotations may still appear as separate entries depending on the Alertmanager version and receiver configuration. Prometheus does not query Alertmanager for active silences or inhibitions, so it continues sending alerts that Alertmanager may suppress.
Documented behavior does not fully specify how Alertmanager reconciles alerts when replicas emit identical label sets but divergent annotation payloads, nor does it clarify whether fingerprint comparison at /api/v2/alerts reliably confirms HA merging across all supported versions.
external_labels configuration across replicas guarantees single-notification semantics without deploying a test rule?35275 reputation · 28 Feb 2021, 10:17 UTC
Alertmanager merges alerts from HA replicas when the label sets are identical, even if annotation values differ. Annotations are excluded from the fingerprint used for deduplication. The deciding factor is whether every replica advertises the same external_labels.
prometheusexternal_labels across replicas produce one fingerprint per alert rule, so notifications are not duplicated.external_labels produce distinct label sets and distinct fingerprints, causing duplicate notifications for the same condition.external_labels block in each replica's Prometheus configuration. Values must match exactly./api/v1/alerts and compare the fingerprint field for alerts originating from different replicas. Matching fingerprints confirm deduplication.Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 28 Feb 2021, 10:16 UTC
Alertmanager’s deduplication fingerprint is a hash of the complete, sorted label set; it does not include annotations, and it is recomputed each time an alert is sent. Because the fingerprint depends only on labels, any change in a label—including those added by service discovery or a differing external_labels value—produces a different fingerprint and causes Alertmanager to treat the alerts as separate groups, even if the underlying rule and annotations are identical. Operators can therefore verify HA alignment by checking that the fingerprint field matches across replicas for the same rule, without needing to inspect annotations.