Using PodDisruptionBudgets to Safeguard Replica Availability During Node Drains

The problem: voluntary disruptions can drop replica counts

When a cluster operator needs to upgrade nodes or scale the cluster in, the kubectl drain command evicts pods from the target node. If the eviction is not constrained, the scheduler may terminate more pods than the application can tolerate, causing the replica count of a Deployment to fall below the desired minimum. This can lead to brief service degradation or failed health checks, especially for stateful workloads that rely on a quorum of replicas.

Thesis: a PodDisruptionBudget (PDB) gives the control plane a safety net

A PodDisruptionBudget defines the maximum number of pods from a replicated application that may be unavailable simultaneously due to voluntary disruptions. By setting minAvailable or maxUnavailable, you tell the API server to block further evictions once the budget would be violated, allowing the drain to proceed only as far as the budget permits.

How a PDB works

• The PDB selector matches the pods of a Deployment, ReplicaSet, or StatefulSet.
• During a voluntary eviction (e.g., node drain, API‑initiated pod deletion), the control plane checks the current number of unavailable pods against the budget.
• If the budget would be exceeded, the eviction is postponed until other pods become available.

Worked example: protecting a three‑replica Deployment

Assume a Deployment named web-frontend with replicas: 3. We want to guarantee that at least two pods stay running during any voluntary disruption.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-frontend-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-frontend

Apply the PDB:

kubectl apply -f web-frontend-pdb.yaml

Verify creation:

kubectl get pdb web-frontend-pdb

Now drain a node that hosts one of the frontend pods (replace <node-name> with the actual node):

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

While the drain runs, watch the pods:

kubectl get pods -l app=web-frontend -w

You should see only one pod transition to Terminating while the other two remain Running. Once the terminated pod is rescheduled elsewhere, the drain completes.

After the operation, inspect the PDB status:

kubectl describe pdb web-frontend-pdb

The output includes a field like Disallowed Pod Disruptions: 0, confirming that the observed evictions stayed within the budget.

Trade‑off: overly tight budgets can block drains

If you set minAvailable: 3 for the same three‑replica Deployment, the PDB would forbid any voluntary eviction. A drain on a node holding a frontend pod would hang until the pod is manually deleted or the PDB is relaxed. This can delay node upgrades, interfere with cluster autoscaler scale‑down, and create a deadlock scenario where no node can be drained.

Practical guidance:

• Start with a budget that reflects the application’s tolerance for unavailable replicas (e.g., minAvailable = replicas - 1 for stateless services).
• Test the behavior in a staging cluster with a dummy drain before applying to production.
• Monitor kubectl describe pdb during maintenance windows to ensure the budget is not being exceeded.

Actionable closing

PodDisruptionBudgets are a lightweight, declarative way to protect replica counts during voluntary disruptions. Define a PDB that matches your workload’s availability requirements, apply it, and verify with kubectl get pdb and kubectl describe pdb before and after drains. Tune the budget based on observed drain times and the criticality of the service, and remember that PDBs do not guard against involuntary failures such as node crashes or pod panics.