Shoot hibernation stuck in Hibernating state due to gardener-controller-manager connectivity loss
0 reputation · 29 Jul 2022, 16:09 UTC
0 reputation · 29 Jul 2022, 16:09 UTC
When Gardener’s controller‑manager loses connectivity to a seed cluster, the Shoot resource is left in the Hibernating condition, even though spec.hibernation.enabled has been set to false. The controller appears to use a cached status that is not refreshed until connectivity is restored.
Identify the mechanism that causes the stale cache to persist, determine how to detect it, and find a configuration or workflow that clears the stale state so the Shoot can transition to Awakening or Ready.
The Gardener release in use does not provide a documented cache‑refresh interval for hibernation reconciliation, and the logs do not indicate a timeout or retry policy for seed‑connection failures.
Gardener’s controller‑manager keeps the last known Hibernation condition in an in‑memory map. When the seed becomes unreachable, the map is never refreshed until the next reconciliation event. Therefore a Shoot can stay stuck in Hibernating even if spec.hibernation.enabled is false.
Shoot.status object).kubectl patch shoot <name> -n <ns> -p '{"metadata":{"annotations":{"hibernation-clear":"$(date +%s)"}}}'Reconciliation triggered by seed status update. Within a few minutes the Hibernation condition should move to Awakening or Ready.No documented configuration or annotation exists that makes the controller poll the seed immediately after reconnection. The only supported method is to trigger reconciliation via a resource change.
kubectl get shoot <name> -n <ns> -o jsonpath='{.status.conditions[?(@.type=="Hibernation")].status}'. A value of Hibernating indicates a stale cache.kubectl get shoot <name> -n <ns> -o yaml and confirm that status.conditions[].type for Hibernation is no longer Hibernating.Do you run more than one replica of gardener-controller‑manager in your control plane? If so, only the replica that last updated the cache will be authoritative, and a patch may need to be applied to each replica’s view.
Use comments to ask for clarification. Post a solution as an answer.
33,775 reputation · 29 Jul 2022, 21:40 UTC
The hibernation condition shown for the Shoot is the last successfully persisted status subresource in the garden cluster, not a private in-memory hibernation cache in gardener-controller-manager.
The controller observes Shoots via standard Kubernetes informers. When the seed cluster is unreachable, reconciliation typically fails before a status write completes, so the previously written hibernation-related conditions remain visible until a later successful reconcile updates them.
Cache invalidation is driven by watch events on the Shoot object in the garden cluster. A metadata or spec change, such as adding gardener.cloud/operation: reconcile, generates a new event and can cause the controller to requeue the object for re-evaluation once seed connectivity is restored, provided the garden API server itself is reachable.