Graceful Shoot Deletion: Hard Timeout vs. Indefinite Wait

Confirmed facts

Gardener v1.80+ exposes deletionGracePeriodSeconds on the Shoot spec. The controller copies this value into the context it passes to the provider SDK for node draining. Today the controller does not enforce a hard timeout; if a provider’s cleanup stalls, deletion proceeds until the provider finishes or the process is terminated externally.

Likely explanation

A hard timeout could be implemented by starting a timer when deletion begins, using a deadline derived from deletionGracePeriodSeconds plus a configurable buffer (e.g., 30 s). When the deadline expires, the controller would cancel the provider‑context and treat the deletion as timed out.

Recommended criteria for aborting a stalled deletion

1. Elapsed deletion time > deletionGracePeriodSeconds + timeoutBufferSeconds (default 30 s).
2. Provider‑context deadline is cancelled, triggering a context.Cancelled error.
3. On timeout, set a new Shoot status condition DeletionTimeoutExceeded with reason Timeout and a message indicating the grace period was exceeded.
4. Stop waiting for the provider; mark the Shoot as Failed (or keep Deleting with the timeout condition) and expose a retryAfterSeconds field suggesting when a retry may be safe.

Interaction with provider‑specific cleanup guarantees

Providers that document bounded cleanup (e.g., AWS ELB deletion within a known window) can safely rely on the timeout; aborting will not leave orphan resources because the provider’s cleanup is guaranteed to finish within the grace period. Providers without such guarantees risk leaving partial resources if the controller aborts mid‑cleanup; in those cases the timeout should be treated as a failure requiring manual cleanup or a retry after the provider’s known maximum duration.

Impact on Gardener’s deletion workflow and API semantics

The controller would add a timeout watchdog before invoking the provider’s cleanup hook. On timeout, it would:

• Cancel the provider context.
• Update Shoot.status.conditions with DeletionTimeoutExceeded.
• Optionally set Shoot.status.lastOperation to Failed.
• Expose a retryAfterSeconds field (or annotation) to guide users when a safe retry can be attempted.

These changes add a deterministic bound to deletion, improve resource reclamation predictability, and introduce a new status condition that users can monitor via the API.

To refine the recommendation, could you share the provider’s typical maximum cleanup duration observed in your environment?