Keeping Replicas Available During Node Drains with Kubernetes Pod Disruption Budgets

Start with the constraint, not the manifest

When you run kubectl drain or a managed node upgrade, Kubernetes can evict several replicas of the same application at once. If your Deployment has three replicas spread across three nodes, a drain that removes all three simultaneously produces an outage even though the cluster is healthy. A Pod Disruption Budget (PDB) is the built-in control that prevents this: it caps the number of Pods that can be simultaneously unavailable because of voluntary disruptions.

The useful takeaway: create a PDB whose selector matches your workload's labels and set minAvailable or maxUnavailable to a value your application can tolerate. The API server will then reject eviction requests that would drop your available replicas below that budget, forcing the drain or upgrade to wait.

Voluntary vs. involuntary disruptions

A PDB only applies when something asks the API server to evict a Pod through the Eviction API. That includes kubectl drain, cluster-autoscaler scale-downs, and managed control-plane or node-pool upgrades. It does not apply to involuntary failures such as node crashes, kernel panics, or OOM kills. If a node dies, its Pods disappear immediately regardless of any PDB. For that class of failure you need redundancy across failure domains—anti-affinity rules, topology spread constraints, and multi-zone scheduling—not a disruption budget.

How the Eviction API enforces the budget

The PDB controller watches Pods that match the budget's label selector and counts how many are available (Ready condition is true). When an evictor submits an Eviction request for one of those Pods, the API server checks every matching PDB. If granting the eviction would push the available count below the budget, the API server returns HTTP 429 (Too Many Requests) or HTTP 500, and the evictor must retry later. A node drain therefore proceeds one Pod at a time when the budget is tight: it evicts a Pod, waits for the replacement to become Ready, then evicts the next.

A worked example: three replicas, one disruption at a time

Assume a stateless web application running three replicas. You can tolerate one replica being disrupted at any moment, but not two. The following Deployment and PDB pair encodes that rule.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.27
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: prod
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web

Apply both manifests from a machine with cluster access (for example, an admin workstation or CI runner) using kubectl apply -f web.yaml. No special permissions beyond namespace edit rights are required for the objects themselves.

With minAvailable: 2 and three healthy Pods, the budget allows exactly one voluntary eviction. If you drain a node that hosts one of these Pods, the drain will evict it and then pause until the Deployment creates a replacement and that replacement reports Ready. Only then will the next eviction be permitted.

minAvailable vs. maxUnavailable

Both fields express the same idea from opposite directions. minAvailable is the floor: the number of Pods that must remain available. maxUnavailable is the ceiling: the number of Pods that may be unavailable at once. For a three-replica workload, minAvailable: 2 and maxUnavailable: 1 are equivalent.

Field	Meaning	Typical use
minAvailable	Minimum available replicas required	Small, fixed-size workloads where you think in terms of quorum or capacity
maxUnavailable	Maximum replicas allowed to be down	Larger replica counts where a percentage is easier to reason about

Both fields accept an integer or a percentage string such as "50%". On Kubernetes 1.21 and newer, use apiVersion: policy/v1. On older clusters the stable API is policy/v1beta1. You can confirm which API groups your cluster serves by running kubectl api-versions | grep policy; you should see policy/v1 on a current cluster.

Limits and common mistakes

Direct Pod deletion bypasses the budget

If you run kubectl delete pod <pod-name>, the API server removes the Pod immediately without consulting any PDB. The Deployment controller will recreate it, but during that window your available count drops. Automation and runbooks should use kubectl drain or the Eviction API rather than raw deletion.

A single replica with a strict budget deadlocks the drain

Setting minAvailable: 1 (or maxUnavailable: 0) on a single-replica workload means no voluntary eviction can ever proceed. A node drain will hang indefinitely because evicting the only Pod would violate the budget. For single-replica services, accept that voluntary disruptions will cause a brief outage, or run multiple replicas so a budget is meaningful.

Overlapping budgets apply the strictest rule

If two PDBs select the same Pods, both are evaluated. The stricter budget wins. For example, if one PDB allows one disruption and another allows zero, the effective limit is zero. Avoid stacking PDBs on the same label set unless you intend that behavior.

Bare Pods have no replacement path

A PDB does not create Pods; it only blocks evictions. If you apply a PDB to bare Pods that are not managed by a Deployment, ReplicaSet, or StatefulSet, an eviction that is eventually allowed will leave that Pod gone permanently. PDBs are designed for controller-managed workloads.

PDBs do not protect against node failure

As noted earlier, involuntary disruptions ignore PDBs entirely. If your three replicas all land on nodes that fail independently, the PDB will not save you. Use topologySpreadConstraints or pod anti-affinity to force replicas across nodes or zones, and treat the PDB as a guardrail for planned maintenance only.

Status fields converge asynchronously

The status stanza of a PDB—currentHealthy, desiredHealthy, and disruptionsAllowed—is updated by the controller, not synchronously with Pod changes. After creating or deleting Pods, allow a few seconds before reading the status.

Verifying the behavior in your cluster

You can confirm the budget is active with a read-only check:

kubectl get pdb web-pdb -n prod -o yaml

Look for disruptionsAllowed: 1 (or the equivalent for your chosen fields) and a selector that matches your Pods. If disruptionsAllowed is 0 when you expect headroom, check that the selector labels exactly match the Pod template labels.

To observe enforcement, pick a node running one of the Pods and run:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

This command requires permissions to evict Pods and update the node (typically cluster-admin or an equivalent role). Run it from an administrative context. The expected behavior is that the drain evicts at most one matching Pod, then waits. If the replacement Pod cannot be scheduled—because of insufficient capacity, taints, or resource quotas—the drain will stall rather than violate the budget. To abort a stuck drain, use Control-C and then kubectl uncordon <node-name> to return the node to schedulable status.

Finally, contrast this with a direct deletion. Run kubectl delete pod <pod-name> -n prod and note that the Pod disappears immediately, demonstrating that the PDB is not consulted for non-Eviction-API deletions.