Stop Node Drains from Evicting Your Entire Kubernetes Workload

You schedule a routine node drain to patch Kubernetes worker nodes. kubectl drain succeeds, but when you check your application, all three replicas are terminating at once. Your load balancer has no healthy endpoints for several minutes. The drain was not broken—you simply never told Kubernetes how many replicas must stay up during voluntary evictions.

A PodDisruptionBudget (PDB) is a Kubernetes object that sets the minimum number—or percentage—of pods that must remain available when an administrator voluntarily evicts pods, such as during a node drain, cluster upgrade, or autoscaler scale-in. It does not protect against a node crashing or a pod failing its own liveness probe; it only gates API-initiated evictions.

What a PDB actually gates

When you run kubectl drain, Kubernetes contacts the Eviction API for each pod on the node. The PDB controller checks whether the eviction would drop the number of available pods below the budget. If it would, the API rejects the eviction and the drain pauses.

PDBs do not apply to:

• Node-pressure evictions initiated by the kubelet
• Pods killed by failed liveness probes
• Involuntary node loss, such as a kernel panic or zone outage
• Resource preemption for higher-priority pods

This means a PDB is a maintenance guardrail, not an availability guarantee. It prevents coordinated downtime during planned work; it cannot prevent surprise loss.

The selector mismatch that protects nothing

A PDB uses spec.selector.matchLabels to find the pods it protects. If those labels do not match spec.template.metadata.labels on the workload's pod template, the budget matches zero pods. Kubernetes does not raise an error. The PDB simply exists with no effect, and a drain can evict every replica.

Before applying a PDB, compare the two label sets directly. A missing environment: production label or a typo like app: payments-v2 versus app: payments silently voids the budget.

Worked example: three replicas, one at a time

Assume a Deployment named payments running three replicas in namespace default. You want at least two pods available during any voluntary disruption. This example uses the policy/v1 API, which is stable in current Kubernetes releases.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments
  template:
    metadata:
      labels:
        app: payments
    spec:
      containers:
      - name: app
        image: payments:1.0

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-pdb
  namespace: default
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments

Apply both objects from a machine with kubectl configured for the cluster. You need permission to create PDBs and Deployments in the namespace; cluster-admin or a namespace-scoped edit role is typically sufficient.

kubectl apply -f deployment.yaml
kubectl apply -f pdb.yaml

Check the budget:

kubectl get pdb payments-pdb -n default

Expected output shows ALLOWED DISRUPTIONS: 1. With three replicas and minAvailable: 2, Kubernetes permits only one voluntary eviction at a time.

Now cordon and drain a node that hosts one payments pod. Replace <node-name> with an actual worker node:

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

The drain evicts one payments pod and then pauses because the budget is exhausted. kubectl get pdb should show ALLOWED DISRUPTIONS: 0 until the replacement pod becomes available. The other two replicas keep serving traffic.

Risk: if you set minAvailable: 3 on a three-replica workload, no voluntary disruption is allowed. A node drain blocks indefinitely until you change the PDB or manually evict pods.

Trade-offs and practical limits

A tight budget protects availability but stalls maintenance. A loose budget lets maintenance proceed but risks losing multiple replicas during a single drain window. There is no automatic middle ground; you choose based on how much concurrent loss your service can absorb.

Also remember that PDBs do not spread replicas across zones or nodes. If all three payments pods happen to run on the same node, a PDB cannot stop that node's drain from taking down the service unless you also use pod anti-affinity or topology spread constraints.

Verify before your next maintenance window

Run this check before scheduling drains:

1. kubectl get pdb --all-namespaces to list every budget.
2. For each PDB, compare spec.selector.matchLabels against the target workload's spec.template.metadata.labels.
3. Confirm minAvailable or maxUnavailable leaves room for at least one node to be drained.

If you need to remove a budget after testing, delete it by name:

kubectl delete pdb payments-pdb -n default

With a correctly aligned PDB, your node drains stop being a gamble. They become a controlled operation where Kubernetes enforces how many replicas can be down at once.