Using Pod Disruption Budgets to Guard Against Unplanned Downtime During Node Maintenance

Problem: Node maintenance can knock out your service

When you upgrade a Kubernetes node or scale the cluster down, the control plane evicts pods from that node via the eviction sub‑resource. If no protection is in place, the eviction proceeds even if it would drop all replicas of a critical deployment, causing a brief outage.

Thesis: A PodDisruptionBudget (PDB) lets you define a safety‑net that blocks voluntary evictions until enough replicas stay available

By attaching a PDB to the selector of your deployment, you tell the API server: "Do not allow an eviction that would leave fewer than X pods running." The control plane checks every matching PDB before honoring an eviction request and postpones the eviction if the budget would be violated.

How a PDB is evaluated

The PDB resource can specify either minAvailable (absolute number or percentage) or maxUnavailable. The controller evaluates the budget against the current count of pods that match the selector, not the desired replica count. This makes the budget resilient to scaling events: if you scale the deployment up, the budget automatically allows more evictions as long as the minimum stays satisfied.

Example definition

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  selector:
    matchLabels:
      app: web
  minAvailable: 3  # at least three replicas must stay up

Place this file in your cluster with kubectl apply -f web-app-pdb.yaml. The selector must match the labels of the pods you want to protect (e.g., a Deployment with app: web).

Worked example: draining a node while a PDB is active

1. Create a test deployment with five replicas:

kubectl create deployment web --image=nginx --replicas=5
kubectl label deployment web app=web

1. Apply the PDB from the example above.

kubectl apply -f web-app-pdb.yaml

1. Check the current state – you should see five pods running:

kubectl get pods -l app=web

1. Identify a node that hosts at least one of the pods, then initiate a drain:

NODE=$(kubectl get pods -l app=web -o wide | head -2 | tail -1 | awk '{print $7}')
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data

When you run the drain, the control plane attempts to evict each pod on $NODE. Because the PDB requires at least three pods to stay available, the eviction is blocked after two pods have been removed (leaving three still running). You will see events like:

Events:
  Type    Reason            Age   From               Message
  ----    ------            ----  ----               -------
  Normal  EvictionFailed    2m    poddisruptionbudget  Cannot evict pod web-...: would violate PodDisruptionBudget web-app-pdb

1. Resume the node after the drain finishes (or cancel it with kubectl uncordon $NODE):

kubectl uncordon $NODE

The deployment controller then recreates the evicted pods to regain the desired five‑replica count.

Trade‑offs and limitations

• Over‑protection: Setting minAvailable too high (e.g., 5 for a 5‑replica deployment) can block node upgrades indefinitely, because the control plane cannot evict any pod without violating the budget. Size the budget with realistic headroom—typically replicas - 1 or a percentage like 80 %.
• Voluntary only: PDBs do not shield against involuntary failures such as node crashes, kernel panics, or container runtime errors. For those scenarios you need replica‑level redundancy, anti‑affinity, or higher replica counts.
• Interaction with other controllers: PDBs work with Deployments, StatefulSets, DaemonSets, and Jobs, but they do not prevent a controller from scaling down below the budget if the scale operation itself is invoked directly (e.g., kubectl scale deployment web --replicas=2). The budget only guards evictions initiated via the eviction API.

Actionable checklist

• Identify critical workloads and label them consistently.
• Define a PDB with minAvailable set to desiredReplicas - 1 (or a percentage that leaves at least one spare).
• Apply the PDB and verify with kubectl describe pdb <name> that the status shows "Allowed eviction: true" when the budget is satisfied.
• Test the behavior in a staging cluster by draining a node and observing the eviction blocking.
• Monitor PDB events during maintenance windows; adjust the budget if you see frequent "Waiting for conditions" messages.

By following these steps you gain a predictable guardrail that lets you perform node upgrades or scale‑down operations without surprising downtime, while still keeping the cluster flexible enough to adapt to changing loads.