Managing Kubernetes Clusters with Gardener Shoot Resources

Problem: You need a repeatable way to create, update, and tear down Kubernetes clusters without manually provisioning control‑plane components.

Gardener solves this by treating each user workload cluster as a Shoot custom resource. The Shoot definition lives in the garden cluster, while the actual control plane runs in a seed cluster. Understanding the lifecycle of a Shoot helps you plan upgrades, avoid downtime, and keep seed resources under control.

How Gardener Shoots Work

A Shoot CR specifies the desired Kubernetes version, provider infrastructure (AWS, Azure, GCP, etc.), and networking. When the Gardener controller reconciles a Shoot:

1. It creates a namespace in the seed cluster named garden-<SHOOT_NAME>.
2. Inside that namespace it deploys etcd (either managed by the provider or embedded) and the control plane components (API server, controller manager, scheduler) via the Machine Controller Manager.
3. Worker nodes are provisioned as MachineDeployments that reference the same Shoot spec.

Updates to the Shoot (e.g., changing spec.kubernetes.version) trigger a controlled rollout: the control plane is upgraded first, followed by worker node machine deployments.

Worked Example: Upgrading a Shoot’s Kubernetes Version

Assume you have a Shoot named production-web running Kubernetes 1.27 and you want to move to 1.28.

Prerequisites

• Access to the garden cluster with cluster-admin role.
• The Shoot’s control plane is configured for high availability (at least three etcd replicas) to avoid API downtime during the upgrade.

Steps

1. Inspect the current Shoot:

   kubectl get shoot production-web -n garden -o yaml
   

   Verify spec.kubernetes.version: 1.27.x and note the seed namespace (garden-production-web).
2. Patch the Shoot to the new version:

   kubectl patch shoot production-web -n garden --type merge -p '{"spec":{"kubernetes":{"version":"1.28"}}}'
   

   This triggers the Gardener controller. No immediate output is guaranteed; you must watch the reconciliation.
3. Monitor the control plane upgrade:

   kubectl logs -n garden -l app=gardener-controller-manager -c manager --follow | grep production-web
   

   Look for log lines indicating that the control plane’s etcd and API server pods are being recreated with the new version.
4. Verify worker node rollout: After the control plane reports Ready, check the MachineDeployments:

   kubectl get machinedeployments -n garden-production-web
   

   You should see a rolling update where old machines are drained and new ones with Kubernetes 1.28 join the cluster.
5. Confirm the Shoot status:

   kubectl get shoot production-web -n garden -o jsonpath='{.status.kubernetesVersion}{ "\n" }'
   

   The output should be 1.28. Additionally, you can run kubectl get nodes inside the Shoot’s kubeconfig to see the node versions.

Risks and Checks

• Permission risk: Patching a Shoot requires garden‑cluster admin rights; misuse could affect other Shoots.
• Verification: After the patch, re‑run the kubectl get shoot command and compare the status.conditions for Progressing and Healthy states.
• Rollback: If the control plane upgrade fails, Gardener will automatically revert the control plane to the previous version. Worker node rollback must be triggered manually by reverting the Shoot spec and re‑applying.

Trade‑offs and Limitations

While Gardener automates cluster lifecycle, there are practical limits:

• Control‑plane availability: If the Shoot’s etcd is a single replica (non‑HA), upgrading the control plane can cause a brief API outage. Ensure HA etcd for production workloads.
• Seed capacity: Each Shoot consumes seed‑cluster resources (etcd, control plane pods, Machine Controller Manager). Creating many Shoots can exhaust CPU/memory in the seed, impacting all managed clusters. Monitor seed resource usage with kubectl top nodes and consider adding more seeds or quotas.
• Etcd backups: Deleting a Shoot removes its etcd backup unless you have explicitly configured a backup bucket in the Shoot’s spec.backup. Verify backup settings before deletion.

Actionable Closing

To safely manage Shoot lifecycles:

1. Always validate HA etcd before version upgrades.
2. Watch Gardener controller logs for reconciliation events; they are the earliest indicator of success or failure.
3. Regularly audit seed namespace count and resource usage to prevent capacity exhaustion.
4. Document backup settings for each Shoot so deletion does not surprise you with lost etcd snapshots.

By treating the Shoot CR as the single source of truth and using the seed cluster as the provisioning engine, you gain repeatable, observable cluster operations while keeping the underlying complexity hidden from application teams.