Prometheus–Alertmanager: how are duplicate alerts with updated annotations reconciled?
0 reputation · 07 Apr 2022, 04:30 UTC
Goal
Determine whether Alertmanager updates the annotations of an alert that Prometheus re‑emits with the same fingerprint but altered annotation values, especially after component restarts.
Constraints and Uncertainty
Alert identity is calculated from the alert name and label set, not from annotations. Alertmanager treats annotations as informational and does not use them for grouping. A restart of either component may cause a new push of an alert that has the same fingerprint but different annotations, leading to potential inconsistencies between what Prometheus exposes and what Alertmanager displays.
Specific Questions
- When Prometheus pushes an alert that has the same fingerprint as an existing one but a changed annotation after a restart, does Alertmanager overwrite the existing annotation or keep the original until the alert resolves?
- How does Alertmanager’s resolve timeout interact with annotation retention in this scenario?
- Is there a documented configuration or API option to force Alertmanager to update annotations on duplicate alerts?