Pre-sync hook Job runs twice when sync retries after transient API server error

Question

SymptomAn ArgoCD Application configured with a pre-sync hook (Job) executes the hook twice when a sync attempt encounters a transient API server error and automatically retries. The first sync attempt creates the hook Job and begins applying manifests; a brief API server outage interrupts the apply. ArgoCD's retry logic re-initiates the sync from the start, creating a second hook Job with the same name before the first Job completes or is cleaned up.ContextArgoCD v2.9+ uses server-side apply (SSA) with field manager argocd for sync operations. The application controller implements exponential backoff for retries (configurable via application.controller.sync.retry.backoff.*), but does not track which hook resources have already been created during a prior attempt. Hook annotations lack built-in idempotency keys, and deletePolicy: HookSucceeded only removes the Job after successful completion — not before a retry spawns a duplicate.ConstraintsHook logic must remain idempotent externally (e.g., writing to a ConfigMap with a timestamp breaks if run twice).Disabling retries is not acceptable; transient errors must be handled automatically.SSA Replace: true cannot be used because it would overwrite concurrent changes from other controllers.What configuration or pattern ensures a pre-sync hook Job runs exactly once per successful sync, even when the sync is retried after a transient failure? Can the retry mechanism be made aware of already-created hook resources? Is there a supported way to gate hook creation on a sync-attempt identifier?

Tasadduq Burney · Answer

When ArgoCD retries a sync after a transient API error, the Application controller re-creates the pre-sync hook Job because no persistent state tracks which hook resources were created during the prior attempt. Server-side apply re-applies the hook manifest with field manager argocd on each retry, and deletePolicy: HookSucceeded only removes the Job after successful completion -- not before a retry spawns a duplicate. ArgoCD has no native annotation or configuration that gates hook creation on a sync-attempt identifier; the retry logic is agnostic of previously created hook resources.

Likely explanation
The retry loop starts from the desired state defined in the Application manifest. Since the hook Job is part of that desired state and lacks a unique sync-attempt identifier, the controller creates a new Job each time it reconciling.

Confirmed facts (ArgoCD v2.9+)

Exponential backoff retries are configurable but do not inspect hook creation history.
deletePolicy: HookSucceeded removes the Job only after it completes successfully.
Field manager argocd is re-applied via SSA on each sync attempt.

Pattern for exactly-once hook execution

Add a unique sync-attempt UID to the Application's metadata.annotations, e.g., argocd.argoproj.io/sync-attempt-uuid.
Configure the pre-sync hook container/image to read this UID and compare it against a tracked ConfigMap or the Job's own annotations from the previous attempt.
If the UID matches the stored value, the hook exits immediately (idempotent no-op).
If the UID is new, the hook proceeds with its work and writes the UID back to the ConfigMap/annotation before exiting.
On the next retry, the UID changes (ArgoCD generates a new attempt ID or you derive it from the sync revision), so the hook runs the intended logic once per sync attempt.

This pattern requires the hook script or image to implement its own idempotency check; ArgoCD does not provide a built-in mechanism to pass a sync-attempt identifier to hook resources.

Missing diagnostic detail: Does the hook script currently embed any state-tracking mechanism (e.g., writing to a ConfigMap, using a file-based lock, or checking annotations)? Confirming this will determine whether the pattern can be applied with minimal changes or if a different approach is needed.

Pre-sync hook Job runs twice when sync retries after transient API server error

Symptom

Context

Constraints

1 answer

Likely explanation

Confirmed facts (ArgoCD v2.9+)

Pattern for exactly-once hook execution

Your perspective belongs here.

0 question comments