Diagnosing Helm Upgrade Timeouts When Using the --atomic Flag

Recognizable condition

Running helm upgrade --atomic hangs or returns an error like timed out waiting for the condition. helm status shows the release in pending-upgrade or failed state, and the upgrade never reaches deployed.

Cause/diagnostic table

Possible cause	What to look for
Insufficient resource quotas causing pod scheduling delays	Pods remain in Pending due to FailedScheduling events.
Slow-to-start Custom Resource Definitions (CRDs)	CRD resources show Establishing condition for longer than expected.
Admission webhook latency or rejection	Events contain AdmissionReview warnings or pods stuck in ContainerCreating with webhook timeouts.
Default Helm timeout (5m) too short for the chart workload	Upgrade fails exactly after ~5m despite resources eventually becoming ready.

Ordered checks

1. Check release state – run helm status -n (requires read access to the namespace). Look for pending-upgrade or failed.
2. Describe failing resources – identify the kind/resource from the status output, then run kubectl describe -n . Check the Conditions and Events sections for clues.
3. Inspect cluster events – execute kubectl get events -n --sort-by=.metadata.creationTimestamp. Filter for Warning events related to scheduling, webhooks, or image pulls.
4. Review pod logs and readiness probes – for any pod that is not Running, run kubectl logs -n and check readiness probe output.
5. Verify the timeout Helm actually used – add --debug to the upgrade command and look for a line like timeout: 5m0s in the output.

Fixes tied to findings

• Increase upgrade timeout – re-run the upgrade with an explicit timeout, e.g.:

  helm upgrade   --atomic --wait --timeout 20m -n 

  (requires permission to create/update releases). If the chart needs more time, adjust the value (e.g., 30m).
• Address resource quota limits – check quota with kubectl describe quota -n . If insufficient, request a quota increase from cluster administrators or reduce resource requests in the chart values.
• Wait for CRDs to be established – before upgrading, run kubectl wait --for=condition=established crd/ or simply delay the upgrade until the CRD shows Established.
• Tune or temporarily disable offending admission webhooks – identify the webhook from events, then either patch its timeout (kubectl patch validatingwebhookconfiguration -p '{"webhooks":[{"timeoutSeconds":30}]}') or, with cluster-admin rights, temporarily exclude the namespace (namespaceSelector) if safe.
• Consider a staged upgrade or disabling --atomic – if the chart creates many pods that start sequentially, you may upgrade sub-charts separately or run helm upgrade --wait --timeout 20m -n (without --atomic) and manually verify readiness afterward. Note: disabling --atomic removes automatic rollback on failure.

Escalation criteria

If the upgrade still fails after raising the timeout to 30m and confirming that:

• no FailedScheduling or webhook warning events appear,
• all CRDs are Established, and
• pod logs show no readiness probe failures,

collect diagnostic data and involve platform support:

1. Run the upgrade with debug logging: helm upgrade --atomic --wait --timeout 30m --debug -n and save the output.
2. Gather Kubernetes API server logs for the time window of the failure (requires cluster-admin access).
3. Open a support ticket with the chart vendor or your platform team, providing the Helm debug log, the API server logs, and the output of kubectl get events -n .

Verification

After applying a fix, rerun the upgrade:

helm upgrade   --atomic --wait --timeout  -n 

Then confirm:

• helm status -n shows STATUS: deployed.
• kubectl get pods -n lists all pods as Running with READY counts matching expectations.
• No new Warning events appear in kubectl get events -n --sort-by=.metadata.creationTimestamp after the upgrade completes.
• If the chart includes a test suite, run helm test -n and verify all tests pass.

Limitations: Increasing the timeout masks underlying scheduling or webhook issues; it does not fix them. Disabling --atomic removes automatic rollback, requiring manual verification and possible manual rollback steps. Changes to quotas or webhooks affect other workloads and should be reviewed by cluster administrators.