Implementing Safe Helm Upgrades with Atomic Rollbacks

A common failure in Kubernetes deployments is the "partial success," where Helm successfully submits manifests to the API server, but the resulting pods fail to start due to image errors or misconfigurations. Without specific safety flags, Helm marks the release as successful because the API request was accepted, leaving the cluster in a broken state.

To prevent this, you can combine helm upgrade --install with the --atomic and --wait flags. This configuration ensures that if the application does not reach a "Ready" state within a defined window, Helm automatically reverts the cluster to the last known stable revision.

Prerequisites

Before implementing atomic upgrades, ensure the following environment requirements are met:

• Helm 3 installed and configured with access to the target Kubernetes cluster.
• Cluster-admin or namespace-specific permissions to create, update, and delete resources.
• A defined readiness probe in your application manifests (Pods/Deployments); without these, --wait cannot accurately determine if the application is healthy.
• Knowledge of the application's average startup time to avoid premature timeouts.

The Atomic Upgrade Procedure

The following command pattern is idempotent, meaning it handles both the initial installation (if the release is absent) and subsequent upgrades (if the release exists) using a single logic path.

# Execute from a CI/CD runner or local terminal with kubeconfig configured
helm upgrade --install [RELEASE_NAME] [CHART_PATH] \ 
  --namespace [NAMESPACE] \ 
  --atomic \ 
  --wait \ 
  --timeout [DURATION]

Flag Definitions

• --install: Instructs Helm to install the chart if it is not already present in the namespace.
• --wait: Blocks the Helm command until all Pods, PVCs, and Services are in a ready state. This is the trigger that allows --atomic to detect a failure.
• --atomic: If the upgrade fails or the timeout is reached, Helm automatically triggers a rollback to the previous single revision. If the initial installation fails, the release is purged.
• --timeout: The maximum duration to wait for readiness. The default is 5m; however, heavy workloads (e.g., Java/Spring Boot) may require 10m or more to avoid spurious rollbacks.

Practical Example: Updating a Microservice

Consider a scenario where you are updating a service named order-processor in the prod namespace. You are deploying a new image tag v2.4.1.

helm upgrade --install order-processor ./charts/order-processor \ 
  --namespace prod \ 
  --set image.tag=v2.4.1 \ 
  --atomic \ 
  --wait \ 
  --timeout 8m

Failure Scenario: If v2.4.1 contains a configuration error causing a CrashLoopBackOff, Helm will wait for 8 minutes. Once the timeout is reached, it will automatically revert the deployment to the previous stable version (e.g., v2.4.0) without manual intervention.

Verification and Diagnostics

After the command completes, verify the release state using the following checks:

1. Confirm Release Status

Check that the release is marked as deployed.

helm status order-processor -n prod

2. Inspect Revision History

Ensure a new revision was created and that no revisions are stuck in pending-upgrade.

helm history order-processor -n prod

3. Verify Workload Readiness

Use kubectl to confirm the actual rollout status of the deployment.

kubectl rollout status deployment/order-processor -n prod

Limitations and Risks

• External Side Effects: Atomic rollbacks only revert Kubernetes manifests. If a pre-upgrade hook executed a database migration that altered a schema, Helm cannot undo those changes.
• CRDs: Custom Resource Definitions (CRDs) are not managed by Helm's upgrade/rollback logic. Changes to CRDs must be handled via a separate lifecycle strategy.
• Hook Persistence: Depending on the delete policy, some hook resources may persist even after an atomic rollback occurs.

Manual Recovery

If --atomic was not used and a deployment fails, you must manually roll back to a specific revision found in the history.

# Roll back to the previous stable revision
helm rollback order-processor [REVISION_NUMBER] -n prod