Managing Kubernetes Versions in Gardener Shoots: Automated Upgrades vs. Version Pinning

The real‑world tension: staying current without breaking workloads

Platform teams that run many tenant clusters face a constant push‑pull: security patches and new Kubernetes features arrive quickly, but production workloads often need a stable version window to avoid surprises. In a multi‑tenant Gardener deployment the Shoot resource is the unit that represents a user‑facing cluster. Its lifecycle is controlled by the Gardener controller running in a Seed cluster, which reconciles the desired state defined in the Shoot spec.

How Gardener handles upgrades automatically

When a Shoot’s spec.kubernetes.version field is set to a version that is newer than the one currently running, the Gardener controller creates an upgrade operation during the Shoot’s configured maintenanceWindow. Outside that window the controller defers the change, which prevents disruptive upgrades during peak traffic. The upgrade itself is performed via the Gardener extension mechanism: a new control plane is provisioned in the Seed, workloads are migrated, and the old control plane is torn down.

To see whether an upgrade is pending or has completed, inspect the Shoot status:

# Run this against the Gardener API server (requires read access to the Shoot namespace)
kubectl get shoot  -n  -o yaml

Look for status.lastOperation; a state: Succeeded with type: Reconcile indicates the last successful reconciliation, while type: Update points to a version change.

Version pinning: locking a Shoot to a specific patch

If you need to keep a Shoot on a exact Kubernetes patch (e.g., 1.27.3) while still receiving Gardener‑managed addon updates, set spec.kubernetes.version to that patch and leave it unchanged. The controller will not automatically move the Shoot to a newer minor version (1.28.x) because the desired version is explicitly pinned. Security patches for the control plane components that are delivered via Gardener addons (like the kube-apiserver etcd backup agent) still apply, as they are reconciled independently of the Kubernetes version.

Pinning introduces two operational trade‑offs:

1. Delayed minor upgrades: moving to a new minor release requires a manual update of the spec.kubernetes.version field, which adds a change‑control step.
2. Maintenance‑window dependency: if the Shoot’s maintenance window is narrow or misaligned with your patch schedule, critical security patches may sit idle until the next window opens.

Worked example: creating a pinned Shoot and verifying its state

Below is a minimal Shoot manifest that pins the cluster to Kubernetes 1.27.3 on the AWS provider, with a maintenance window of 02:00‑04:00 UTC.

apiVersion: gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: example-pinned-shoot
  namespace: garden
spec:
  provider:
    type: aws
    workers:
    - name: worker
      machineType: t3.medium
      minReplicas: 2
      maxReplicas: 3
      volumeSize: 20Gi
  kubernetes:
    version: 1.27.3
  maintenance:
    window:
      start: "02:00"
      end:   "04:00"
  networking:
    pods: "10.250.0.0/16"
    services: "10.100.0.0/16"
    nodes: "10.0.0.0/16"

Apply the manifest with:

kubectl apply -f example-shoot.yaml

After the Shoot transitions to the Ready state, verify that the version is pinned:

kubectl get shoot example-pinned-shoot -n garden -o jsonpath='{.status.kubernetesVersion}'

The output should be 1.27.3. To confirm that no automatic minor upgrade will occur, you can temporarily change the desired version to a newer patch (e.g., 1.27.4) and observe that the controller does not initiate an upgrade until the next maintenance window.

Limitation and practical check

The primary limitation of version pinning is that you must manually intervene to adopt a new minor Kubernetes release. This can increase operational overhead, especially in large fleets where many Shoots are pinned to different versions.

A practical way to monitor pinning compliance across a Garden is to run a simple label‑based query:

# Lists all Shoots whose current Kubernetes version differs from the spec version
kubectl get shoot -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,SPEC_VERSION:.spec.kubernetes.version,STATUS_VERSION:.status.kubernetesVersion

Any row where the two version columns differ indicates a Shoot that is either undergoing an upgrade or has drifted from its intended pin.

Actionable takeaway

For multi‑tenant platforms, let Gardener’s automated upgrade mechanism handle routine patching within maintenance windows, and use version pinning only for workloads that truly require a fixed Kubernetes release. Regularly audit the status output shown above to catch any Shoots that have missed their upgrade window or are unintentionally running a version different from the spec.