Prometheus–Alertmanager: how are duplicate alerts with updated annotations reconciled?

Annotation Reconciliation Behavior

When Alertmanager receives an alert with a fingerprint (the hash of the alert name and its labels) that matches an existing active alert, it overwrites the existing annotations with the new values. Because annotations are treated as informational metadata and are not part of the alert's identity, Alertmanager updates the alert object in place whenever a duplicate is pushed from Prometheus.

Interaction with Resolve Timeout

The resolve_timeout does not prevent annotation updates; it only determines when an alert is marked as resolved if Prometheus stops sending it. The interaction works as follows:

• Within the timeout window: Any duplicate alert received with the same fingerprint will immediately update the annotations of the firing alert.
• After the timeout expires: Alertmanager marks the alert as resolved and removes it from the active set. If Prometheus later re-emits the alert (e.g., after a restart), it is treated as a brand-new alert instance with a fresh set of annotations.

Configuration and API Options

There is no specific configuration flag or API option to toggle this behavior. Overwriting annotations on duplicate fingerprints is the default, hard-coded behavior of Alertmanager. To change how alerts are grouped or routed, you must modify the labels, as Alertmanager ignores annotations for all routing, silencing, and inhibition logic.

Verification Steps

To verify this behavior in your environment (assuming Alertmanager v0.10+), you can perform the following test:

1. Create a Prometheus alert rule with a static label set and a dynamic annotation (e.g., a description containing a version number).
2. Verify the alert is firing in the Alertmanager UI or via curl http://<alertmanager>:9093/api/v2/alerts.
3. Modify the annotation value in the Prometheus rule and reload the Prometheus configuration.
4. Observe that the Alertmanager API reflects the updated annotation immediately, while the alert remains in the "firing" state without creating a second entry.

Assumptions and Constraints

This explanation assumes the label set is identical. If a single label is added, removed, or changed during the Prometheus restart, Alertmanager will generate a new fingerprint, resulting in two distinct alerts rather than an update to the existing one.

Diagnostic Detail Needed: If you are seeing duplicate alerts instead of updates, please provide the exact label sets of the "original" and "updated" alerts to verify if the fingerprints actually match.