Zero‑downtime Shoot upgrades with Gardener Control Plane Migration

Problem: Zero‑downtime upgrades are hard

Traditionally, upgrading a Gardener Shoot cluster meant draining the existing control plane, deleting it, and provisioning a new one with the target Kubernetes version. During that window the API server is unavailable, which can cause timed‑out requests, failed deployments, and interrupted workloads. Operators therefore schedule upgrades during maintenance windows or accept brief outages.

Thesis: Control Plane Migration removes the downtime

Gardener 1.5 introduced Control Plane Migration (CPM). When enabled, Gardener brings up a second control plane running the desired Kubernetes version alongside the existing one. API traffic is gradually shifted from the old to the new plane, and once the shift is complete the old plane is torn down. The result is an upgrade that keeps the API server reachable throughout the process.

How CPM works

At a high level CPM follows these steps:

1. Gardener creates a new control plane in the same seed, using the version specified in spec.kubernetes.version.
2. Both control planes run concurrently; the seed’s API‑server load balancer (or equivalent) starts sending a small fraction of traffic to the new plane.
3. The traffic share is increased incrementally while Gardener monitors health checks on both planes.
4. When the new plane handles 100 % of traffic, the old plane is scaled down and deleted.
5. The Shoot’s status.kubernetesVersion is updated and the migration condition is set to ControlPlaneMigrationSuccessful.

Prerequisites

• Gardener version ≥ 1.5 (check with gardenerctl version).
• A seed cluster with sufficient CPU, memory, and etcd storage to host two control planes temporarily.
• Recent etcd backup of the Shoot (recommended before starting CPM).
• The Shoot must not have any ongoing maintenance operation that conflicts with CPM.

Worked example

Below is a minimal Shoot manifest that enables CPM and targets Kubernetes 1.29.0. Placeholders like <seed-name> and <namespace> should be replaced with your actual values.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-shoot
  namespace: 
spec:
  seedName: 
  kubernetes:
    version: 1.29.0
  maintenance:
    controlPlaneMigration:
      enabled: true
  # … other spec fields (provider, networking, etc.) …

Apply or update the Shoot:

gardenerctl shoot update --shoot my-shoot --seed 

After the command is issued, watch the Shoot’s status:

kubectl get shoot my-shoot -n  -o yaml

Look for the condition:

status:
  conditions:
  - type: ControlPlaneMigrationSuccessful
    status: "True"
    reason: MigrationCompleted
    message: "Control plane migration finished successfully"
  lastOperation:
    state: Succeeded
    description: "Control plane migration completed"

During the migration you can verify API availability with a simple loop, for example:

while true; do kubectl get pods -n kube-system --request-timeout=2s && sleep 5; done

If the loop continues to return results without errors, the API server remained reachable.

Trade‑offs and limits

• Resource usage: Two control planes run side‑by‑side, roughly doubling CPU, memory, and etcd write load for the duration of the migration. Ensure the seed has spare capacity; otherwise scheduling of control‑plane components may fail.
• Operational complexity: If the migration aborts (e.g., due to a node failure), you may need to manually clean up the old or new control plane and retry. Keeping a recent etcd backup reduces risk.
• Version requirement: CPM is unavailable in Gardener < 1.5; older releases will simply ignore the controlPlaneMigration.enabled flag.
• Seed suitability: Not all seed providers expose the necessary load‑balancer hooks for traffic shifting; verify with your seed’s documentation.

Actionable closing

1. Test CPM in a non‑production seed first. Deploy a throw‑away Shoot with CPM enabled and monitor the shoot.gardener.cloud/control-plane-migration condition.
2. Set up an alert on the migration condition staying in Progressing for longer than a predefined timeout (e.g., 30 minutes).
3. Validate resource consumption during the test (seed node CPU/memory, etcd latency) to confirm you have headroom for production.
4. After successful validation, roll out CPM‑enabled Shoot updates to production, always preceded by an etcd backup and a quick sanity check of the Shoot’s status.lastOperation.

By following these steps you can take advantage of Gardener’s Control Plane Migration to keep your Shoot clusters available while staying up‑to‑date with the latest Kubernetes releases.