Argo CD Self-Heal Reconciliation Boundary with Server-Side Apply Idempotency

Answer: How Argo CD treats a self‑heal retry

Argo CD decides whether a self‑heal cycle is a no‑op or a full reconcile by comparing the cached desired manifest (the version stored in its internal state, sourced from Git) with the live Kubernetes state it observes via the API. The decision is made before any server‑side apply call is issued.

Confirmed facts

• If the cached manifest and the live state are deep‑equal (field‑by‑field, including annotations and labels that Argo CD tracks), Argo CD sets operationState: SYNCED. The next self‑heal tick sees no drift and therefore performs no reconciliation work – the server‑side apply is never invoked, or if invoked it receives an empty patch and becomes a no‑op thanks to Kubernetes’ idempotency.
• If any field differs – even a field that is not managed by Argo CD’s apply‑prune strategy – the comparison yields a non‑empty strategic merge patch. Argo CD then updates operationState to OUT_OF_SYNC. On the following self‑heal tick it executes a full reconcile: it recomputes the patch and issues a server‑side apply call. Because server‑side apply is idempotent, re‑applying the same patch does not cause duplicate writes, but the reconcile still runs.
• The transition from an inert retry to a full reconcile therefore hinges solely on the presence of a detected drift** between the cached desired manifest and the live state at the moment the self‑heal loop evaluates the application.

How operationState tracking influences the outcome

Argo CD persists operationState in the application’s status (status.sync). This field is:

• Set to SYNCED after a successful sync when the cached and live states match.
• Set to OUT_OF_SYNC whenever a drift is detected during a sync or self‑heal evaluation.
• Can be manually altered (e.g., argocd app set --sync-option Prune=false or clearing the status) – doing so can temporarily suppress a full reconcile until the next external change causes a new drift detection.

Thus, adjusting operationState does not change the underlying comparison logic; it only changes the starting point for the next self‑heal evaluation. If you force operationState: SYNCED while the live state has drifted, the next self‑heal will still detect the mismatch and transition to OUT_OF_SYNC before deciding to reconcile.

Conditions for full reconcile vs. no‑op retry

• No‑op retry: Live state equals cached desired manifest (including all fields Argo CD considers). operationState is SYNCED. Self‑heal finds zero‑size patch → skips apply.
• Full reconcile: Any field difference exists (e.g., an external controller adds/removes an annotation, changes a label not managed by Argo CD, or modifies a field covered by a different apply‑prune rule). operationState becomes OUT_OF_SYNC. Self‑heal runs the reconcile path, computes a patch, and invokes server‑side apply.

Practical verification steps (version‑aware)

1. Confirm you are using Argo CD ≥ v2.4, where server‑side apply is enabled by default:

   argocd version --client
   argocd version --server

2. Enable debug logging for the application controller (e.g., set --loglevel debug in the controller deployment) and watch the logs during a self‑heal interval.
3. Look for lines such as:
• operationState: SYNCED – indicates a no‑op self‑heal.
• operationState: OUT_OF_SYNC – indicates drift detection.
• patch size: 0 bytes vs. patch size: N bytes – shows whether a non‑empty patch was generated.
4. To test the boundary, apply an external change to a field Argo CD does not manage (e.g., kubectl annotate pod/foo bar=baz) and observe that the next self‑heal logs OUT_OF_SYNC and a non‑zero patch. Reverting the annotation should bring the next self‑heal back to SYNCED with a zero‑size patch.

Missing diagnostic detail

The exact set of fields Argo CD includes in its equality check depends on the ignoreDifferences configuration and any custom jsonPath exclusions. If you have non‑default ignore rules, please confirm whether the externally modified field is covered by those rules, as that determines whether the change triggers drift.