Inconsistent alert deduplication across mesh members

Direct answer

Traefik Mesh v2.9 does not use any label or internal identifier to deduplicate alerts across mesh members, because the mesh data plane does not perform alert deduplication. Alert generation, grouping, and silencing are owned by Prometheus and Alertmanager. Model knowledge contains no verified mesh-level grouping label, and no confirmed Traefik Mesh v3.0 deduplication scope has been established. Reported behavioral changes between 2.9 and 3.0 most likely reflect changes to exported metric labels in the underlying Traefik Proxy or telemetry exporter, not a new mesh grouping criterion.

Confirmed facts vs. likely explanation

Confirmed: Traefik Mesh is in maintenance/deprecation status and does not document a mesh-level alert deduplication engine. Alertmanager receives alerts scraped from mesh members and groups them according to its own configuration.

Likely explanation: In v2.9 deployments, each mesh member exports telemetry with per-instance identifiers such as instance, pod, or kubernetes_pod_name. Alertmanager therefore sees separate alert instances per member unless group_by is normalized. If a v3.0 rollout appears to alter deduplication, the probable cause is a change in those exported labels rather than a mesh feature.

Steps to complement mesh-level behavior

Because the mesh does not deduplicate, configure Alertmanager to collapse repeats while preserving distinct incidents:

1. Inspect the labels present on alerts received from each mesh member using the Alertmanager API or UI.
2. Define an Alertmanager route with group_by on service-level identity labels—typically ['alertname', 'service', 'job']—and deliberately exclude per-member identifiers such as instance, pod, or kubernetes_pod_name.
3. Set group_wait and group_interval to batch transient repeats without delaying genuine incidents.
4. Use match or match_re on silence rules to target the service-level labels, never on the per-member identifiers alone.

Verification before rollout

Because label names vary by exporter and alert rule, verify the actual label set in staging:

amtool alert --alertname=YourAlert --verbose
# or query the Alertmanager API:
curl -s http://alertmanager:9093/api/v2/alerts | jq '.[0].labels'

Compare the label maps from a v2.9 member and a v3.0 member. If the only difference is the presence or absence of instance/pod labels, the grouping change is telemetry-side, not mesh-side.

Missing diagnostic detail

Which Prometheus job label and exporter version are attached to your mesh-member alerts? This determines whether group_by: ['alertname', 'service'] is sufficient or whether you must first relabel per-member identifiers in Prometheus before scraping.