Architecture Note: Provisioning Kubernetes Clusters with Gardener Shoot Resources

The Problem: Decoupling Cluster Lifecycle from Infrastructure

Managing multiple Kubernetes clusters across different cloud providers often leads to fragmented tooling and inconsistent configurations. The challenge is to maintain a centralized control plane that can provision, upgrade, and delete clusters (Shoots) without manually managing the underlying virtual machines or API servers for every single environment.

Takeaway: By using Gardener, you shift from managing individual clusters to managing Shoot custom resources. The Gardener controller handles the heavy lifting of deploying the control plane into a Seed cluster and the worker nodes into the target cloud infrastructure.

Requirements

To implement this architecture, the following components must be present in the Garden cluster (the root management cluster):

• Seed Cluster: At least one functional Seed cluster. This is a specialized Kubernetes cluster that hosts the control planes of the Shoots.
• Project: A Project resource that provides a namespace for isolation and access control for your Shoots.
• SecretBinding: A resource that links the Project to a Kubernetes secret containing cloud provider credentials. This ensures credentials are not hardcoded in the Shoot specification.

Smallest Suitable Design

The most efficient starting design utilizes a single Seed cluster to manage multiple Shoots. In this model, the Garden cluster acts as the orchestrator, while the Seed cluster acts as the hosting environment for the Shoot's API server, scheduler, and etcd.

Each Shoot resource defines the following essential parameters:

• Kubernetes Version: The specific version of Kubernetes to deploy.
• Infrastructure Provider: The cloud type (e.g., AWS, Azure, GCP) and the target region/zone.
• Worker Pools: Machine types, volume sizes, and scaling limits (min/max nodes).
• Networking: CIDR ranges for pods, services, and nodes.

The Gardener controller-manager continuously reconciles the current state of the infrastructure against the desired state defined in the Shoot resource.

Trust and Data Boundaries

To minimize the blast radius of a security breach, Gardener separates the management and execution planes:

• Garden Cluster: The highest trust zone. It holds the Gardener API and the SecretBinding credentials.
• Seed Cluster: A medium trust zone. It runs the Shoot control plane components. It does not have permanent access to the Garden cluster's root secrets; credentials are injected only as needed for the specific Shoot.
• Shoot Worker Nodes: The lowest trust zone. These run the actual workloads and have no access to the Seed or Garden clusters.

Operational Checks

To verify the health of a Shoot and its reconciliation loop, use the following diagnostic steps:

1. Check Operation State

Run this command in the Garden cluster to see if the Shoot is currently processing a change:

kubectl get shoot <shoot-name> -n <project-namespace> -o jsonpath='{.status.lastOperation.state}'

Expected result: The state should move from Processing to Succeeded. If it stays in Processing for an extended period, check the status.conditions field for errors.

2. Monitor Controller Logs

If a Shoot is stuck, inspect the Gardener controller-manager logs in the gardener namespace:

kubectl logs -n gardener -l app=gardener-controller-manager

3. Verify Seed Health

Ensure the hosting Seed is healthy, as a Seed failure impacts all Shoots it hosts:

kubectl get seed <seed-name> -o yaml

Failure Modes

Conditions That Change the Design

• Data Residency: If laws require the control plane (etcd) to reside in a specific country, you must deploy a Seed cluster in that region and explicitly assign the Shoot to it via spec.seedName.
• Private Networking: For environments where public API endpoints are forbidden, set spec.networking.apiServerVisibility to Private. This requires established VPC peering or VPN connectivity between the Garden and Seed clusters.

Example Shoot Configuration

The following manifest creates a minimal Kubernetes 1.27 cluster on AWS. Run this using kubectl apply -f shoot.yaml with permissions to create resources in the specified project namespace.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: production-cluster
  namespace: project-alpha
spec:
  kubernetes:
    version: 1.27.0
  provider:
    type: aws
    region: eu-central-1
    workers:
      - name: general-purpose
        machineType: t3.medium
        volumeType: gp2
        volumeSize: 20
        autoScalerMin: 2
        autoScalerMax: 5
        zones:
          - eu-central-1a
          - eu-central-1b
  networking:
    pods: 10.250.0.0/16
    services: 10.251.0.0/16
    nodes: 10.252.0.0/16
    apiServerVisibility: Public
  addons:
    - name: nginx-ingress
      enabled: true

Verification and Limitations

To verify the result, poll the Shoot status until it reaches Succeeded, then attempt to retrieve the kubeconfig for the Shoot to ensure the API server is responsive.

Limitations:

• Infrastructure Only: Gardener manages the cluster lifecycle, not the applications. You must use a separate GitOps tool (like ArgoCD or Flux) for workload delivery.
• Version Drift: CRD fields change between Gardener releases. Always verify the v1beta1 or v1 API version against your specific installation.
• Provider Variance: Not all cloud providers support every feature (e.g., specific autoscaling behaviors). Check the CloudProfile for your provider.

Rollback: To undo the operation, delete the Shoot resource: kubectl delete shoot production-cluster -n project-alpha. Gardener will trigger the cloud provider to delete the worker nodes and remove the control plane from the Seed cluster.