maxSurge=0 vs maxSurge=1 for fixed-capacity RollingUpdate with readiness probes
0 reputation · 30 Aug 2020, 18:39 UTC
0 reputation · 30 Aug 2020, 18:39 UTC
A small application is being migrated with a Kubernetes Deployment using RollingUpdate. The goal is to keep the Service available during the rollout while the cluster has a fixed capacity ceiling that cannot be exceeded.
With maxSurge=0 the controller must terminate an old pod before a new one can be scheduled, which respects the resource ceiling but temporarily reduces the number of Ready pods that can serve traffic. With maxSurge=1 an additional pod can be created, shortening rollout time and preserving serving capacity, at the cost of transient resource contention if no spare headroom exists.
The unresolved behavior concerns readiness probe failures during the rollout. A new pod that does not become Ready stays out of Service endpoints, and the interaction with maxSurge and maxUnavailable determines whether capacity drops while waiting for readiness. The Deployment controller does not automatically roll back on readiness failure.
Which maxSurge setting preserves availability under a fixed capacity constraint when new pods may experience readiness lag? Does the controller retain old pods when new pods never become Ready with maxSurge=0 versus maxSurge=1? Is there a documented guarantee about endpoint availability in the window between old pod termination and new pod readiness?
35275 reputation · 30 Aug 2020, 23:47 UTC
Under a hard fixed capacity ceiling neither setting guarantees uninterrupted Service availability with readiness lag. maxSurge=1 with maxUnavailable=0 preserves the Ready pod count by keeping old pods until new pods become Ready, but it requires one pod of spare capacity. maxSurge=0 respects a hard ceiling by terminating an old pod before creating a new one, and it guarantees a window with fewer Ready pods while the new pod waits for readiness.
If a new pod never becomes Ready, old pods are retained only with maxSurge=1/maxUnavailable=0. With maxSurge=0 the old pod is already terminated and is not restored; the Deployment stays below desired Ready replicas until manual intervention. There is no controller-level guarantee of endpoint availability in the window between old pod termination and new pod readiness. A pod is removed from Service endpoints when it enters Terminating, and the new pod is added only after its readiness probe succeeds.
With a true hard ceiling where no additional pod can be scheduled, maxSurge=1 will cause the surge pod to stay Pending. Pending surge pods do not reduce existing availability, they only slow the rollout. If the ceiling is enforced by a ResourceQuota or a hard limit that blocks Pending pods from being created, maxSurge=1 will block progress entirely.
With maxSurge=0 the rollout respects the ceiling but you accept a temporary reduction in Ready pods equal to maxUnavailable while the new pod is initializing and passing readiness probes. The length of that window is controlled by initialDelaySeconds, periodSeconds and failureThreshold of the readiness probe, plus minReadySeconds if set.
kubectl get endpoints and kubectl rollout status deployment and inspect Deployment conditions for ProgressDeadlineExceeded.One missing diagnostic detail that changes the recommendation: is the fixed capacity ceiling a hard quota that prevents creation of a Pending pod, or is it a scheduling headroom limit where a Pending surge pod is allowed to wait without evicting existing pods? If it is the latter, maxSurge=1 with maxUnavailable=0 is preferable for availability even with a ceiling.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.