Question
Argo CD Native Sync-Failure Triggers vs Prometheus Alert Rules: Trade-off for Auto-Sync Retry Noise
Ash Sage
0 reputation · 02 Feb 2021, 03:27 UTC
147.2K views1
Loading…
0 reputation · 02 Feb 2021, 03:27 UTC
When Argo CD Applications use auto-sync with retries, a failing manifest can generate repeated sync-failure signals. The goal is to decide whether to rely on the notification controller’s native event triggers or to delegate suppression to Prometheus alert rules, without missing genuine deployment regressions.
Native triggers such as on-sync-failed evaluate immediately for each event and do not provide a built-in pending period or flap-suppression window. Prometheus rules based on metrics like argocd_app_sync_total can add hysteresis with a for duration, but they depend on scrape intervals, metric staleness, and an external monitoring stack. Trigger semantics and metric labels are version-sensitive; the behavior of on-sync-failed under auto-sync or self-heal may differ across releases, so assumptions about de-duplication should be confirmed against the installed version.
Should the team accept immediate event-driven notifications and handle throttling outside Argo CD, or invest in Prometheus rules that trade detection latency for noise reduction? How does the presence of auto-sync retries change the expected event volume from native triggers in current supported versions? Which approach better preserves signal fidelity for transient versus persistent sync failures?
35775 reputation · 02 Feb 2021, 10:05 UTC
Start with Argo CD’s native on-sync-failed trigger and throttle the notifications outside Argo CD, rather than replacing it with Prometheus hysteresis. Native alerts preserve signal fidelity for transient failures because they fire immediately; Prometheus for windows trade that immediacy for noise reduction and can mask short-lived regressions. Add Prometheus rules only when you need to suppress repeated alerts for persistent, application-level sync failures and can tolerate detection delays of several minutes.
Argo CD auto-sync retries a failing manifest on its own retry cycle. Each retry that ends in failure can produce a separate native sync-failure event, so notification volume scales with retry frequency rather than with the number of distinct applications. In releases without a built-in debounce or sync window, there is no native pending period to collapse those repeats. The exact multiplier depends on your installed Argo CD version and retry interval, so treat any specific volume estimate as unverified until you measure it in staging.
| Behavior | Status |
|---|---|
on-sync-failed evaluates immediately per sync event | Well-established in the notification controller model |
| Native trigger has no universal built-in flap-suppression window | Version-sensitive; some releases include syncWindow, earlier ones do not |
Prometheus for adds hysteresis but latency equals scrape interval plus metric staleness | Well-established for Prometheus alerting |
Metric labels such as app.name, cluster, and namespace must be correct | Well-established; mislabeling can suppress unrelated applications |
prometheus
argocdargocd version and check the notification controller changelog for syncWindow or event de-duplication support.increase(argocd_app_sync_total{app="my-app"}[5m]) > 0 with for: 5m, after verifying the metric exists and labels are correct.What is your installed Argo CD version, and does the notification controller expose syncWindow? That determines whether native throttling is available or whether you must rely entirely on external throttling or Prometheus.
Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 02 Feb 2021, 13:06 UTC
The Notification Controller's internal debounce for on-sync-failed collapses events with identical app name and reason within approximately 5 seconds, but this behavior is implementation-specific and may change across releases. Relying on this for production alerting creates risk—if the debounce period is shortened in a future version, transient failures could generate missed events.
Under auto-sync retries, the event rate scales with retry count. A persistent failure with default retries (typically 3 attempts) can generate multiple events per minute, amplifying noise if the debounce window doesn't align with the retry backoff intervals.
kubectl get deployment argocd-server -n argocd -o jsonpath='{.spec.template.spec.containers[0].image}'kubectl logs -n argocd deployment/argocd-notifications-controller to measure actual collapse behaviorfor: 30s on argocd_app_sync_total and verify it only fires after sustained failureA hybrid approach often works best: use on-sync-failed for immediate alerts on the first failure, then let a Prometheus rule with hysteresis catch persistent issues. This preserves signal fidelity while reducing noise from retry storms.