Configuring Control-Plane High Availability in Gardener Shoots

Ensuring Control-Plane Resilience

In Gardener, the control plane of a Shoot cluster is hosted on a Seed cluster. If the Seed's infrastructure suffers a node or zone failure, the Shoot's API server and etcd quorum can be compromised, leading to cluster instability or total loss of management access. To prevent this, you must explicitly configure the failureTolerance setting.

The primary takeaway is that control-plane HA is independent of worker-node HA. Setting your worker nodes to span multiple zones does not automatically protect the control plane; you must configure spec.controlPlane.highAvailability.failureTolerance to ensure the Seed-side components are distributed across failure domains.

Understanding failureTolerance

The failureTolerance field determines how Gardenlet (the Gardener controller) renders the control-plane components on the Seed cluster. There are two primary levels of tolerance:

• node: Ensures that control-plane replicas are distributed across different nodes. This protects against a single VM failure but not a data center outage.
• zone: Ensures that replicas are spread across different availability zones. This protects against the loss of an entire zone.

When zone is selected, Gardenlet applies pod anti-affinity and topology spread constraints to the kube-apiserver and etcd pods. This ensures that the main and events etcd quorums remain functional even if one zone becomes unavailable.

Worked Configuration Example

Below is a Shoot specification that implements both control-plane HA (at the zone level) and worker-node HA. This configuration assumes the underlying CloudProfile supports multiple zones.

spec:
  controlPlane:
    highAvailability:
      failureTolerance: zone
  provider:
    workers:
    - name: pool-1
      minimum: 3
      maximum: 5
      # Worker HA: Spreading nodes across zones
      zones: ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
      machine:
        type: "m5.large"
        image:
          name: gardenlinux
          version: "v1.2.0"

Verification and Diagnostics

Because the control plane resides on the Seed, you cannot verify its HA status from the Shoot's own kubeconfig. You must have access to the Seed cluster with permissions to view the shoot's specific namespace.

Step 1: Verify the Shoot Spec

Run this on the Gardener Core cluster to ensure the specification was accepted:

# Run on Gardener Core
kubectl get shoot <shoot-name> -n garden-<project-name> -o yaml

Check that spec.controlPlane.highAvailability.failureTolerance is set to zone and the status shows no failing conditions.

Step 2: Verify Pod Distribution on the Seed

Switch context to the Seed cluster and inspect the pods in the shoot's namespace (usually garden-<project>-<shoot>):

# Run on the Seed Cluster
kubectl get pods -n <shoot-namespace> -o wide

Cross-reference the node names with the node labels to ensure topology.kubernetes.io/zone differs across the kube-apiserver and etcd replicas.

Step 3: Verify etcd Quorum

Confirm that the etcd stacks (managed by Druid) have the expected member count:

# Run on the Seed Cluster
kubectl -n <shoot-namespace> get etcd -o wide

Limitations and Common Pitfalls

The "False Security" Gap

A common mistake is assuming that zones: [...] in the worker pool configuration provides control-plane HA. This is incorrect. The worker zones only affect where your application workloads run. If you have 10 workers across 3 zones but failureTolerance is unset or set to node, a single zone failure on the Seed could still take down your entire cluster's API.

Operational Risks during Updates

Changing the failureTolerance value triggers a rolling update of the kube-apiserver and etcd components. This can cause brief API unavailability. To mitigate this:

• Schedule the change within the spec.maintenance.timeWindow.
• Coordinate the change with worker node updates; avoid simultaneous mass-reboots of workers to ensure you don't lose both API access and workload capacity at once.

Resource Constraints

Enabling zone tolerance increases the number of replicas for control-plane components. Ensure the Seed cluster has sufficient CPU and memory capacity to handle the additional pods, as this may increase cloud costs for the Seed infrastructure.

Rollback Procedure

If the HA configuration causes instability or exceeds seed resource limits, revert the failureTolerance value to node or remove the highAvailability block entirely.

# Update the Shoot spec to revert
kubectl patch shoot <shoot-name> -n garden-<project> --type merge -p '{"spec":{"controlPlane":{"highAvailability":{"failureTolerance":"node"}}}}'

Gardenlet will then reconcile the Seed-side components, reducing the replica count and removing the zone-specific anti-affinity constraints.