Controlling Pod Termination with Kubernetes PDBs During Node Drains

Problem: Aggressive Pod Termination During Maintenance

\n

When you run kubectl drain to upgrade a node or shift workloads, Kubernetes can terminate pods faster than is healthy for your application. Without guidance, a single node drain could ripple through your cluster, leaving insufficient replicas to serve traffic.

\n

Thesis: Pod Disruption Budgets Give You Control

\n

Pod Disruption Budgets (PDBs) let you specify a minAvailable or maxUnavailable threshold per workload. The Kubernetes eviction logic respects that threshold during voluntary disruptions such as node drains, cluster autoscaling events, or manual edits.

\n

What PDBs Guarantee (and What They Don't)

\n

\n
• PDBs operate at the workload level. After a disruption, at least minAvailable pods must remain running.
\n
• They only apply to voluntary disruptions triggered by kubectl drain, cluster autoscaling, or resource edits. They do not protect against evictions from PodSecurityPolicy removal, kube-system pod lifecycle events, or preemption-driven termination.
\n
• Setting minAvailable to 0 effectively disables the safety net, allowing complete workload drain.
\n

\n

Configuring minAvailable and maxUnavailable

\n

A typical PDB for a deployment with three replicas, allowing at most one pod to be evicted at a time:

minAvailable: 2 maxUnavailable: 1

\n

Diagnostic Command

\n

To inspect an existing PDB and verify its fields, run:

kubectl get poddisruptionbudgets -A -o yaml

\n

Where to run: Any machine with kubectl configured to talk to the target cluster.

\n

Required permissions: core/v1/poddisruptionbudgets.get (or list) RBAC role. In most default setups, authenticated users can list PDBs across namespaces.

\n

Meaningful placeholders: Replace -A with -n namespace to scope a single namespace, or target a specific budget with kubectl get poddisruptionbudget budget-name.

\n

Expected checks: Look for the minAvailable and maxUnavailable fields under the spec section. Confirm that minAvailable is greater than 0 if you intend to prevent drain, and that the value aligns with your workload's replica count.

\n

Relevant risks: A misconfigured minAvailable: 0 allows the entire workload to be drained, contradicting the availability guarantee. If your cluster uses pod priority and preemption, PDBs may interact poorly, potentially starving lower-priority pods during resource pressure.

\n

Version-Dependent Behavior and Cloud Provider Defaults

\n

Kubernetes 1.21 and newer releases include more stable PDB status fields, making it easier to verify which pods are available versus disrupted. Older releases may lack these refined fields, so upgrading is advisable if you rely on PDB status for automation.

\n

Managed clusters such as GKE and EKS often pre-create PDBs for system workloads (for example, DNS, monitoring agents). These default PDBs cannot be deleted, only modified. If you're testing ad-hoc drain scenarios, verify whether a PDB is provider-enforced before attempting removal.

\n

Trade-Off: Preemption, Priority, and the minAvailable=0 Trap

\n

Setting minAvailable too low or to 0 defeats the purpose of the budget. Beyond misconfiguration, PDBs interact with the scheduler's preemption logic: if a pod has a low priority, the scheduler may preempt it first, but if a PDB protects that pod, the eviction may be deferred, creating a starvation scenario under sustained pressure.

\n

Practical verification: after applying a PDB, simulate a drain on a node and watch pod lifecycles. Use kubectl get pods -w alongside kubectl drain --dry-run=client to observe behavior without immediate termination.

\n

Actionable Closing

\n

\n
1. Inspect your workload's PDB: kubectl get poddisruptionbudgets -A -o yaml. Confirm minAvailable reflects your availability goal.
\n
2. If no PDB exists, create one aligned with your replica count. Example:

   apiVersion: v1 kind: PodDisruptionBudget metadata: name: web-frontend-pdb namespace: production spec: minAvailable: 2 selector: matchLabels: app: web-frontend

\n
3. Apply it: kubectl apply -f web-frontend-pdb.yaml
\n
4. Verify the PDB is active: kubectl get poddisruptionbudget web-frontend-pdb -n production -o yaml
\n
5. Simulate a drain on a test node: kubectl drain --ignore-daemonsets --delete-emptydir-data --dry-run=client and observe that pods respect the minAvailable threshold before terminating.
\n

\n

Remember: PDBs are a safety net, not a guarantee against all failures. Pair them with health checks and careful rollout strategies for production resilience.