Maintaining Availability During Node Drains: Choosing a Pod Disruption Budget Strategy

The Problem: Unplanned Downtime During Planned Maintenance

\n

When you perform a cluster upgrade or scale down a node pool via a cluster autoscaler, Kubernetes triggers a “voluntary disruption.” By default, the kubectl drain command attempts to evict all pods from a node. Without a defined strategy, Kubernetes may evict too many replicas of a critical service simultaneously, causing a temporary outage or performance degradation despite having a Deployment in place.

\n

The goal is to ensure that a minimum number of healthy pods remain operational throughout the eviction process, preventing the infrastructure layer from accidentally taking down the application layer.

\n\n

Comparing Availability Strategies

\n

While standard replica counts ensure availability during involuntary failures (like a hardware crash), they do not prevent the API server from evicting multiple pods during voluntary maintenance. You must decide between using a Pod Disruption Budget (PDB) or relying solely on replica distribution.

\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Strategy	Mechanism	Protection Scope	Risk
Replica Scaling	Increasing replicas count	Involuntary failures (node crash)	Node drains can still evict all pods if they reside on one node.
PDB: minAvailable	API server rejects evictions if count drops below X	Voluntary disruptions (upgrades, drains)	Can block node drains if set too high.
PDB: maxUnavailable	API server rejects evictions if X pods are already down	Voluntary disruptions (upgrades, drains)	May allow too many pods to drop if total count is low.

\n\n

Trade-offs and Decision Logic

\n

Choosing between minAvailable and maxUnavailable depends on whether your priority is a guaranteed floor of capacity or a controlled ceiling of loss.

\n\n

Use minAvailable when:

\n

\n
• You have a strict minimum capacity requirement to handle baseline traffic.
\n
• You are using a percentage (e.g., minAvailable: 50%) to scale the budget automatically as you scale your Deployment replicas.
\n

\n\n

Use maxUnavailable when:

\n

\n
• You want to ensure that only a small fraction of your fleet is offline at once, regardless of the total size.
\n
• You are managing a large-scale cluster where losing a fixed number of pods (e.g., maxUnavailable: 1) is safer than a percentage.
\n

\n\n

Critical Constraint: The Deadlock Risk

\n

Setting minAvailable equal to your total replica count (e.g., 3 replicas, minAvailable: 3) creates a deadlock. Because the API server will refuse to evict any pod that would bring the count below 3, kubectl drain will hang indefinitely. This prevents cluster upgrades and requires manual deletion of the PDB or pods to resolve.

\n\n

Implementation Example

\n

Consider a service with 3 replicas. To ensure at least 2 pods are always running during a node upgrade, apply the following PDB. This requires cluster-admin permissions or a Role with create/patch permissions on policy/v1.PodDisruptionBudget.

\n\n

# Define the PDB targeting pods with the label 'app=api-gateway'
cat <

\n\n

Validating the Budget

\n

To verify the PDB is functioning, you can simulate a node drain. Run these commands on a worker node hosting your pods:

\n\n

\n
1. Check current status: Run kubectl get pdb api-gateway-pdb. Ensure ALLOWED DISRUPTIONS is 1 (since 3 are running and 2 are required).
\n
2. First Eviction: Run kubectl drain --ignore-pod-eviction. The first pod should be evicted successfully.
\n
3. Second Eviction: Attempt to drain another node hosting a remaining pod. The API server should return an error stating that the eviction would violate the Pod Disruption Budget.
\n

\n\n

Limitations

\n

\n
• Involuntary Failures: PDBs provide zero protection against kernel panics, power loss, or hardware failure. They only govern the Eviction API.
\n
• Controller Dependency: A PDB only works if a controller (like a Deployment or StatefulSet) is present to recreate the pod on a different node. If you use a standalone Pod, the PDB will block the drain, but the pod will never be replaced.
\n
• Version Assumption: This guide assumes Kubernetes v1.21+ where policy/v1 is the stable API for PDBs.
\n

\n\n

Rollback

\n

If a PDB is blocking critical infrastructure maintenance, remove the budget to allow evictions to proceed:

\n

kubectl delete pdb api-gateway-pdb