Perform a Safe Rolling Update of a Kubernetes Deployment

Desired outcome

Update the container image of an existing Deployment to a newer version while keeping the application available. Kubernetes will create a new ReplicaSet and gradually replace old pods, respecting the update strategy you define.

Prerequisites

• A Kubernetes cluster running version 1.16 or later.
• kubectl configured to communicate with the cluster (you need at least edit rights on the target namespace).
• The Deployment you want to update already exists.
• The new container image is pushed to a registry that the cluster can pull from (image pull secrets configured if the registry is private).

Procedure

1. Identify the Deployment and container

   Run:

   kubectl get deployments -n <namespace>

   Note the Deployment name (<deployment>) and the exact container name you want to update (<container>).

2. Set the new image

   Use the imperative command (or edit the manifest and apply). Example updating an NGINX container:

   kubectl set image deployment/<deployment> <container>=nginx:1.23 -n <namespace>

   This tells Kubernetes to create a new ReplicaSet with the specified image.

3. Monitor the rollout

   Watch the progress:

   kubectl rollout status deployment/<deployment> -n <namespace>

   The command blocks until all new pods are Ready and the old ReplicaSet is scaled to zero.

Expected checks

1. Verify rollout completion

   After the status command returns, confirm:

   kubectl get deployment <deployment> -n <namespace> -o wide

   The IMAGE column should show the new tag (e.g., nginx:1.23) for all replicas.

2. Check pod readiness

   Ensure every pod is Ready:

   kubectl get pods -l app=<deployment> -n <namespace>

3. Validate service traffic

   If the Deployment backs a Service, send a request to the Service endpoint (or use an ingress) and confirm the response reflects the new version (e.g., a version header or changed HTML).

Recovery options

If the new version causes errors, you can revert to the previous ReplicaSet:

1. Undo the rollout

   kubectl rollout undo deployment/<deployment> -n <namespace>

   Kubernetes scales down the faulty ReplicaSet and scales up the previous one.

2. Manual rollback (optional)

   Identify the old ReplicaSet:

   kubectl get rs -n <namespace>

   Then scale it up and delete the new one:

   kubectl scale rs <old-replicaset> --replicas=<desired> -n <namespace>
   kubectl delete rs <new-replicaset> -n <namespace>

Limitations and practical verification

• Strategy limits: Ensure maxSurge and maxUnavailable in the Deployment’s strategy are set so that the cluster never exceeds resource quotas or drops below the minimum required pods. You can view them with:

kubectl get deployment <deployment> -n <namespace> -o yaml | grep -A2 strategy

• PodDisruptionBudget (PDB): A tight PDB may block evictions during the update. Check PDBs in the namespace:

kubectl get pdb -n <namespace>

• Image pull issues: If the rollout stalls, inspect events:

kubectl describe deployment <deployment> -n <namespace>

Look for Failed or ErrImagePull reasons and verify image pull secrets or node connectivity.

After a rollback, repeat the verification steps to confirm the old image is running across all pods.