Fine‑grained health‑check probes in Gardener shoots after version upgrade

Question

When upgrading a Gardener managed Kubernetes environment from a release that only provides basic liveness and readiness probes to a newer release that exposes fine‑grained probe configuration (e.g., separate startup, liveness, readiness settings with adjustable timeouts and failure thresholds), operators must decide whether to retain the legacy probe behavior or adopt the new granular options for existing shoot clusters.
The goal is to maintain consistent health‑check semantics across all shoots while avoiding unexpected restarts or prolonged unavailability caused by mismatched timeout values. Constraints include the provider‑agnostic nature of shoot YAML, the requirement that seed cluster quotas remain sufficient, and the need to keep the same manifest usable across AWS, GCP, and Azure without provider‑specific overrides.
Because the change introduces new fields that are ignored by older control planes, there is uncertainty about how the Gardener controller treats missing fine‑grained sections after an upgrade and whether default values align with the previous basic probe behavior.
What are the recommended steps to evaluate the impact of enabling fine‑grained probes on existing shoots? How can operators verify that the new probe settings do not degrade recovery latency or trigger false‑positive failures? Is there a safe migration path that allows a gradual rollout of the granular configuration without requiring a full shoot recreation?

Muntasir · Answer

Answer first: in a Gardener shoot you cannot change per-container liveness/readiness/startup probes via the shoot spec itself. Probes live in workload Pod specs. What a Gardener control-plane upgrade can change is the defaults and the kubelet / control-plane component health-check configuration that Gardener applies to the shoot. Because of that, enabling "fine-grained probes" is not a shoot-wide toggle; it is a change to defaults and/or to how you render workload manifests.
Confirmed facts vs likely explanation
Confirmed: Gardener shoot YAML is provider-agnostic. Shoot spec fields are reconciled by the Gardener controller; unknown fields are ignored by older control planes. A shoot upgrade does not recreate the shoot, and provider-specific overrides break portability across AWS, GCP and Azure.
Likely explanation, needs verification: The new release adds separate startupProbe, livenessProbe and readinessProbe defaults for control-plane components and/or kubelet extraConfig. Missing fine-grained sections are ignored by older control planes, so upgraded shoots keep legacy probe behavior until the new fields are explicitly set in the shoot spec. The exact mapping of those fields to container probes is version-dependent and not documented here.
Minimal evaluation steps for this case

Identify scope. Confirm whether you are changing kubelet health-check defaults, control-plane component probes, or default Pod probes injected via an admission controller. The migration path differs.
Review release notes for the target Gardener version. Note which probe fields were added and their default timeoutSeconds, periodSeconds, failureThreshold.
Extract current effective values from a representative shoot. Example read-only checks:
kubectl get shoot <shoot-name> -o yaml | yq '.spec.kubernetes.kubelet, .spec.provider.workers[].kubelet'
kubectl -n kube-system get pods -l component=kube-apiserver -o yaml

Compare legacy vs new defaults field by field. Flag any decrease in timeoutSeconds or increase in aggressiveness of failureThreshold * periodSeconds.
Test in an isolated seed. Clone the shoot spec to a dev seed, apply the fine-grained settings only to kubelet extraConfig or the relevant component, and observe pod startup time and restart rate under normal load.

Verification that recovery latency is not degraded

Monitor restart counts and readiness gates: kubectl get pods -w and compare to baseline.
Check controller logs for warnings about unknown fields or probe application.
Validate effective kubelet configuration on nodes and that observed timeout and period match the intended values.
Run a controlled failure test against a health endpoint and measure detection latency ≈ failureThreshold * periodSeconds.

Safe gradual rollout without shoot recreation
A shoot can be patched in place. If the change is kubelet-related, update shoot.spec.kubernetes.kubelet or shoot.spec.provider.workers[].kubelet.extraConfig and let Gardener roll the worker machines. If the change is workload defaults, update the source manifests or the admission configuration and roll Deployments incrementally. Keep the manifest provider-agnostic and avoid provider-specific fields.
One missing diagnostic detail changes the recommendation: are you referring to kubelet/control-plane health-check defaults managed by Gardener, or to default container probes applied to user workloads via an admission controller? Confirming that determines whether the change is made via shoot spec or via workload rendering.

Fine‑grained health‑check probes in Gardener shoots after version upgrade

1 answer

Confirmed facts vs likely explanation

Minimal evaluation steps for this case

Verification that recovery latency is not degraded

Safe gradual rollout without shoot recreation

Your perspective belongs here.

0 question comments