Maintaining Service Availability During Kubernetes Node Drains with PodDisruptionBudgets

The Availability Gap During Cluster Maintenance

When performing voluntary disruptions—such as node drains for kernel upgrades, cluster autoscaling, or version migrations—Kubernetes evicts pods to clear a node. Without constraints, the eviction API can terminate too many replicas of a stateless service simultaneously, causing a spike in latency or a total outage for users.

The goal is to ensure a latency-sensitive service maintains a minimum capacity floor during these maintenance windows without over-provisioning resources or stalling cluster upgrades indefinitely.

Choosing Your Availability Strategy

To protect a service, you must define a PodDisruptionBudget (PDB). This is a policy that limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions. For a deployment with 5 replicas requiring at least 3 to be ready to meet latency SLOs, you have three primary configuration paths.

Trade-offs and Constraints

minAvailable is the safest choice for strict SLOs. It counts only pods that have passed their readiness probes. If your application has a long startup time or is experiencing image pull errors, a minAvailable setting can block kubectl drain operations entirely, requiring manual intervention or a --force flag (which bypasses the PDB).

maxUnavailable is often preferred for CI/CD pipelines where rollout speed is a priority. It defines the "budget" for failure. However, if your Horizontal Pod Autoscaler (HPA) scales the deployment down to 3 replicas, a maxUnavailable: 2 setting would allow the cluster to drop to 1 replica during a drain, potentially violating your latency requirements.

Implementation: Protecting a 5-Replica Deployment

This example assumes Kubernetes v1.21+ using the policy/v1 API. We will implement a PDB for a service labeled app: latency-sensitive-api.

1. Define the PDB

Run the following command or apply the YAML manifest in the namespace where your deployment resides. This requires permissions to create poddisruptionbudgets in the policy API group.

cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-availability-pdb
  namespace: production
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: latency-sensitive-api
EOF

2. Verify the Budget Status

Check that the PDB has correctly identified the pods. If the ALLOWED DISRUPTIONS column is 0, it means the current number of ready pods is already at or below your minAvailable threshold.

# Run in the cluster admin context
kubectl get pdb api-availability-pdb -n production

3. Validate via Simulated Drain

To test the protection, attempt to drain a node hosting one of these pods. Use the --ignore-daemonsets flag to avoid errors from system pods.

# Run on a node containing the target pods
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
If you attempt to drain multiple nodes simultaneously such that the remaining ready pods would drop below 3, the eviction API will return a 429 Too Many Requests error, and the kubectl drain command will hang or fail until a new pod becomes Ready on a different node.
Critical Limitations

Involuntary Disruptions: PDBs do not protect against hardware failure, kernel panics, or node crashes. They only affect the Eviction API.
Selector Mismatch: If the selector.matchLabels in the PDB does not exactly match the labels on the Pods, the PDB will be created but will control zero pods, providing no actual protection.
Readiness Dependency: A PDB is only as effective as your readiness probes. If a pod is running but the probe is failing, the PDB treats it as unavailable, which may block cluster upgrades.
Rollback
To remove the availability constraint and allow unrestricted node drains, delete the PDB resource:
kubectl delete pdb api-availability-pdb -n production