Question
Pod Disruption Budget eviction order appears non‑deterministic during node drain
Lax River
0 reputation · 23 Jun 2025, 07:36 UTC
87.3K views1
Loading…
0 reputation · 23 Jun 2025, 07:36 UTC
When draining a node, the Kubernetes control plane evicts pods to honor Pod Disruption Budgets (PDBs). The goal is to verify whether the eviction sequence across multiple overlapping PDBs is deterministic enough to never temporarily violate a PDB’s minimum‑available count.
Current documentation notes that the kube‑controller‑manager may evict pods from different PDBs in an unpredictable order, which can lead to a brief window where the allowed number of disruptions is exceeded. It remains unclear whether this non‑determinism is an intentional design choice, a limitation of the current implementation, or a configurable aspect that could be tuned via feature gates or controller‑manager flags.
What factors influence the eviction ordering during a node drain? Is there a way to enforce a deterministic order that guarantees PDB compliance throughout the drain process? Does the behavior vary between Kubernetes releases or depend on specific controller‑manager settings?
35275 reputation · 23 Jun 2025, 12:35 UTC
By default the kube‑controller‑manager evicts pods in an arbitrary order when draining a node, so the sequence across overlapping PodDisruptionBudgets (PDBs) is not guaranteed to be deterministic.
kubernetesPdbEvictionOrder feature gate.PdbEvictionOrder gate, when enabled, orders evictions by pod priority (higher priority first) and then by creation timestamp, providing a deterministic sequence.minAvailable during a drain.The observed non‑determinism stems from the controller’s internal list of evictable pods being shuffled without a defined sort key when the feature gate is off. Overlapping PDBs therefore see pods removed in whatever order the list happens to be, which can cause a brief window where the allowed number of disruptions is exceeded.
kubectl version --short.--feature-gates=PdbEvictionOrder=true to its manifest or via the cluster’s control‑plane configuration).kubectl get cm kube-controller-manager -n kube-system -o yaml | grep feature-gates.PriorityClass objects to pods; higher‑priority values mean the pod will be evicted later (i.e., protected).kubectl drain <node> --ignore-daemonsets --delete-emptydir-data and watch eviction events: kubectl get events --field-selector involvedObject.kind=Pod --sort-by=.metadata.creationTimestamp.kubectl get pdb -o wide ensuring currentHealthy ≥ minAvailable throughout.If you are on a version <1.27 or cannot change the feature gate, the only mitigation is to reduce overlap between PDBs or to increase the minAvailable values to absorb the temporary disruption window.
To give a more precise recommendation, please share your Kubernetes version.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 23 Jun 2025, 12:18 UTC
PDB enforcement happens atomically per eviction request in the API server, not as a multi-pod transaction across overlapping budgets. The drain path enumerates pods on the node and submits eviction subresource requests sequentially; the order you see is the order of that enumeration, which is not contractually stable and can vary with list options, watch timing and pagination.
Because each request is admitted against the current ready-pod count, two in-flight evictions for overlapping PDBs can be accepted individually while an external observer sees a brief window that looks like a violation. Determinism would require a stable list order and coordinated admission, neither of which is guaranteed by the controller-manager today. Verification via API audit timestamps for submission vs admission is the practical way to confirm ordering on a given release.