Question
Argo CD Notifications and Slack: Filtering Transient Health States
Lax Vale
0 reputation · 13 Mar 2025, 00:45 UTC
100.5K views1
Loading…
0 reputation · 13 Mar 2025, 00:45 UTC
Managing alert volume in a large-scale Argo CD deployment requires balancing visibility with notification noise. The Argo CD Notifications controller allows for trigger definitions based on application status changes, but default configurations often emit redundant messages during the reconciliation process.
The goal is to ensure that notifications are only dispatched for terminal or actionable states, specifically avoiding alerts while an application is in a Progressing health state during a standard sync operation. This requires a precise configuration of the when clause within the trigger definition to differentiate between a legitimate failure and a transient state change.
when expression be structured to suppress notifications during the Progressing state while still capturing a transition to Degraded?Synced notification is sent only after a period of instability, rather than for every minor reconciliation loop?33275 reputation · 13 Mar 2025, 02:20 UTC
1. When expression to suppress Progressing but still fire on Degraded
argocd
kubernetestriggers:
- name: app-health
when: "{{ or (eq .health.status \"Degraded\") (eq .health.status \"Suspended\") }}"
template: slack-alert
slack:
channel: '#argo-alerts'
The when clause above matches only the two terminal health states that you care about. Because it does not reference Progressing, any transition into that transient state will not fire the trigger.
2. Final Synced notification only after stability
triggers:
- name: app-sync
when: "{{ and (eq .sync.status \"Synced\") (eq .health.status \"Healthy\") }}"
delay: 5m
template: slack-sync
slack:
channel: '#argo-sync'
The delay: 5m line tells the controller to wait five minutes after the condition becomes true before sending the message. If another sync loop occurs during that period, the timer resets, ensuring the message only appears after a stable state persists for the full delay.
when and conditions blocks. A syntax error will silently drop the trigger.argocd app get <app> -o yaml | grep health.status.delay option is supported in Argo CD v2.7+; older releases will ignore it.Open the notifications ConfigMap:
kubectl -n argocd edit configmap argocd-notifications-cm
Add the two trigger blocks shown above under the triggers section.
Save and exit. The controller will reload the ConfigMap automatically.
Trigger a change that causes a Progressing state (e.g., a rolling update) and confirm no Slack message is posted.
Induce a Degraded state (e.g., delete a pod with a non‑ready status) and verify a Slack alert appears.
Apply a change that results in a stable Synced state and watch for the Slack message after the 5‑minute delay.
argocd app get <app> -o yaml | grep health.status to see the current health state.argocd-notifications-controller logs for parsing errors: kubectl logs -n argocd deploy/argocd-notifications-controller | grep error.To fine‑tune the delay value or confirm supported syntax, please let me know the exact Argo CD version you are running.
Use comments to ask for clarification. Post a solution as an answer.
5,330 reputation · 13 Mar 2025, 03:11 UTC
Instead of listing the terminal states you want to keep, you can explicitly exclude the transient Progressing state while still allowing any other health value to fire. This approach automatically captures future states (e.g., a new Stalled value) without editing the trigger.
triggers:
- name: app-health
when: "{{ ne .health.status \"Progressing\" }}"
template: slack-alert
slack:
channel: '#argo-alerts'
The ne (not equal) operator evaluates to true for Healthy, Degraded, Missing, Suspended, and any other status Argo CD may introduce.
The delay timer is reset each time the when condition changes. If you want the notification to send only after the application remains Synced and Healthy for a continuous window, combine the delay with a condition that also checks the operation phase:
triggers:
- name: app-sync-stable
when: "{{ and (eq .sync.status \"Synced\") (eq .health.status \"Healthy\") (eq .operationState.phase \"Terminated\") }}"
delay: 10m
template: slack-sync
slack:
channel: '#argo-sync'
Here the trigger fires only when the sync has finished (operationState.phase == \"Terminated\") and the health is healthy. The 10‑minute delay ensures that a brief fluctuation back to Progressing during the window resets the timer, preventing premature notifications.