Stop node drains from emptying your Kubernetes replicas

The maintenance drain that took down quorum

You cordon a node and run kubectl drain to apply a kernel patch. Kubernetes evicts every pod on that node. Your three-replica deployment briefly runs one pod. If that workload is a database or a controller needing quorum, one replica is not enough. Requests fail until new pods schedule elsewhere.

The useful takeaway: Kubernetes does not protect running replicas during voluntary disruptions by default. A Pod Disruption Budget (PDB) sets a minimum number of healthy pods that must stay up during voluntary events like node drains and cluster upgrades. It does not stop a node crash or kernel panic, and it cannot create capacity that does not exist.

What a PDB controls, and what it does not

A PDB names a label selector and chooses one of two budgets:

• minAvailable: the minimum number of pods that must remain healthy. With three replicas and minAvailable: 2, at most one pod can be evicted at a time.
• maxUnavailable: the maximum number of pods that may be down. With three replicas and maxUnavailable: 1, the effect is the same as minAvailable: 2.

The API server blocks voluntary evictions that would breach the budget. Involuntary loss—node hardware failure, kernel panic, zone outage—is outside a PDB's scope. PDBs also do not move pods, reschedule them, or guarantee they land on different nodes.

Version assumption: this guide targets Kubernetes 1.21 or later, which serves PDBs under policy/v1. The older policy/v1beta1 was deprecated in 1.21 and removed in 1.25.

Worked example: budget a three-replica deployment

Suppose you run a deployment named cache-tier with three replicas and label app: cache-tier. Create this file as cache-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cache-tier-pdb
  namespace: default
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: cache-tier

Apply it from a machine with kubectl configured and namespace edit permission:

kubectl apply -f cache-pdb.yaml

Check that the budget is active:

kubectl get pdb cache-tier-pdb -n default

Inspect ALLOWED DISRUPTIONS, CURRENT HEALTHY, and DESIRED HEALTHY. The status fields are eventually consistent, so re-read the object before assuming an eviction is permitted.

Testing the drain on a non-production node

To observe enforcement, cordon a test node and attempt a drain. Replace <node> with your node name. This command evicts pods, so run it only on a non-production node or a staging cluster:

kubectl cordon <node>
kubectl drain <node> --delete-emptydir-data --ignore-daemonsets

If the budget is active and only one pod can be spared, further evictions may be blocked until a replacement pod becomes healthy. To return the node to service:

kubectl uncordon <node>

Required permissions: cluster-admin or a role allowing evict on pods and cordon/uncordon on nodes. Risk: a drain removes workloads from the node; do not run it on the only node hosting critical system components without a replacement plan.

Limitations and overlapping budgets

Two PDBs can match the same pods. When that happens, Kubernetes enforces the most restrictive budget. Two teams might each set minAvailable: 2 on three replicas, which sounds safe, but together they can block every eviction because neither side will yield its pod.

Setting minAvailable equal to the replica count is another common mistake. It prevents any voluntary eviction until a replacement pod can schedule elsewhere. If the cluster is full, your node drain waits indefinitely.

A PDB is not high availability. Pair it with adequate replica counts, pod anti-affinity, and topology spread across zones so replacement pods actually have somewhere to run.

Verify before you depend on it

Confirm API support:

kubectl api-versions | grep policy

Inspect all PDBs in a namespace:

kubectl get pdb -n default -o wide

For a specific budget, check allowedDisruptions, currentHealthy, and desiredHealthy in the status block. If allowedDisruptions is zero, voluntary evictions are blocked until healthy replicas rise above the floor.

Closing

Use a PDB when maintenance or autoscaling might evict pods faster than your application can absorb. Start with minAvailable set to one less than your replica count, test on a staging node, and monitor kubectl get pdb before and after the drain. It guards the window during voluntary disruption; it does not replace replicas, zones, or a real availability plan.