Preventing Downtime During Maintenance with Kubernetes Pod Disruption Budgets

When you upgrade a Kubernetes node, patch a kernel, or allow the cluster autoscaler to remove underutilized hardware, you use the kubectl drain command. This tells Kubernetes to evict pods so they can be rescheduled elsewhere. Without a safety net, a drain might evict all replicas of a critical service simultaneously, resulting in a total outage during routine maintenance.

The solution is the Pod Disruption Budget (PDB), a practical engineering control that ensures a minimum number of pods remain available during these voluntary disruptions. It acts as a gatekeeper for the eviction API, blocking any request that would drop your capacity below a specified threshold.

Voluntary vs. Involuntary Disruptions

It is critical to understand what a PDB actually protects. PDBs only govern voluntary disruptions. These are planned actions initiated by humans or automated systems, such as:

• Node drains for maintenance
• Cluster autoscaler scale-in events
• Rolling updates of Deployments

PDBs do not protect against involuntary disruptions. If a node crashes, a pod runs out of memory (OOM), or a network partition occurs, Kubernetes will kill the pods immediately regardless of your PDB settings. A PDB is a planning tool, not a disaster recovery mechanism for hardware failure.

Defining Your Budget

A PDB uses a label selector to identify a group of pods. You then define the budget using one of two fields, but never both in the same object:

• minAvailable: Specifies the minimum number or percentage of pods that must remain running.
• maxUnavailable: Specifies the maximum number or percentage of pods that can be down simultaneously.

For most high-availability applications, a common pattern is maxUnavailable: 1. This ensures that even during a node drain, only one pod is removed at a time, leaving the rest to handle traffic.

Example: Three-Replica Web Service

In this scenario, we have a Deployment with three replicas. We want to ensure that at least two replicas are always online during maintenance.

Step 1: Create the Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx

Step 2: Create the Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-server-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web

Step 3: Verify with a Drain

If you run kubectl drain <node-name> --ignore-daemonsets, the Kubernetes eviction API will attempt to remove the pods. If one pod is already being evicted, the PDB controller will block the second request until the first pod is healthy and rescheduled. You can monitor the status using:

kubectl get pdb web-server-pdb

The Deadlock Trade-off

The most common failure mode with PDBs is the "deadlock." If you have a Deployment with only one replica and a PDB with minAvailable: 1, the pod can never be evicted because evicting it would violate the budget. This causes kubectl drain and autoscaler operations to hang indefinitely.

Additionally, be cautious with percentages. If you have 3 replicas and set maxUnavailable: 50%, Kubernetes rounds up the number of allowed disruptions, meaning 2 could be evicted, leaving only 1 running. Always use absolute numbers for small-scale sets to ensure predictable behavior.

Actionable Takeaways

• Pair every critical Deployment or StatefulSet (with ≥2 replicas) with a PDB using maxUnavailable: 1.
• Verify PDB status before and after maintenance with kubectl get pdb and kubectl describe pdb to see currentHealthy and desiredHealthy.
• Test drains in a staging cluster to confirm that voluntary evictions respect the budget and that the autoscaler can still scale in.
• Remember that PDBs do not protect against node crashes or OOM kills; design your services to tolerate involuntary disruptions separately.