Automating etcd Backup and Restore in Gardener Shoot Clusters

Problem: protecting etcd without manual snapshots

When a Shoot cluster’s etcd database becomes corrupted or lost, the API server stops responding and workloads cannot be scheduled. Manual etcd snapshots are easy to forget, and ad‑hoc scripts often miss the correct retention settings, leaving operators exposed to data loss or unexpected storage charges.

Thesis: Gardener’s built‑in etcd backup sidecar automates snapshot creation and restore, letting you define schedules and retention in the Shoot spec.

How Gardener’s etcd backup sidecar works

When Gardener creates a Shoot, it injects a sidecar container into the etcd StatefulSet. The sidecar watches a Backup custom resource (CR) that you define in the Shoot spec. On the schedule you provide (cron expression), the sidecar runs etcdctl snapshot save, streams the snapshot to the object‑storage bucket you configured, and tags it with a timestamp.

Configuring backup and retention

Backup behavior is controlled under .spec.backup in the Shoot manifest:

spec:
  backup:
    schedule: "0 2 * * *"   # every day at 02:00 UTC
    retention:
      maxBackups: 30
      garbageCollection:
        enabled: true
    destinations:
      - bucket: "my-gardener-backups"
        region: "eu-central-1"
        provider: "s3"

The schedule field accepts standard cron syntax. maxBackups limits how many snapshots are kept; older ones are removed when garbage collection is enabled. You can add multiple destination entries for cross‑region replication.

Worked example: creating a backup and restoring from it

1. Create the Shoot with backup config – apply a Shoot manifest that includes the .spec.backup block above. Ensure you have cluster-admin rights on the Garden cluster.
2. Verify the sidecar is running – kubectl -n garden‑<shoot-name> get pods -l etcd-sidecar=true should show a pod ready.
3. Wait for the first snapshot – either let the cron trigger or manually create a Backup CR:

kubectl -n garden-<shoot-name> create -f - <

Check the object storage (e.g., AWS CLI):

aws s3 ls s3://my-gardener-backups/<shoot-uid>/ --recursive

You should see a file like etcd-snapshot-2026-09-15T02-00-00Z.db.

4. Simulate an etcd failure – delete the etcd pod for member 0:

kubectl -n garden-<shoot-name> delete pod etcd-0

The API server will become unavailable for a short period.

5. Trigger a restore – create a Restore CR pointing to the latest snapshot:

kubectl -n garden-<shoot-name> create -f - <

Gardener launches a temporary etcd pod, loads the snapshot, and replaces the failed member. During this time the API server is down; plan the restore inside a maintenance window.

6. Validate recovery – after the Restore CR reaches Ready state, run:

   kubectl get --raw='/readyz?timeout=5s'
   

   If the response is ok, the API server is healthy. Then check that workloads are rescheduled:

   kubectl get pods -A | grep -v Completed | grep -v Running
   

   Any Pending pods should transition to Running as the controller manager reschedules them.

Trade‑offs and limitations

• Storage cost – retaining many snapshots can inflate object‑storage bills. Monitor bucket size with cloud‑provider metrics and adjust maxBackups or lifecycle rules accordingly.
• Restore downtime – replacing an etcd member requires a brief API‑server outage. For workloads that cannot tolerate any interruption, consider deploying a standby control plane (Gardener’s highAvailability feature) or performing restores during a scheduled maintenance slot.
• Network bandwidth – snapshots are streamed to the bucket; large etcd datasets may saturate limited links, especially when using cross‑region replication. Test backup duration in a non‑production Shoot before committing to a tight schedule.

Actionable next steps

1. Add a .spec.backup block to your existing Shoot manifests, starting with a conservative schedule (e.g., weekly) and a low maxBackups value.
2. Enable object‑storage lifecycle rules to expire snapshots older than your retention period, providing a safety net against runaway costs.
3. Document a restore run‑book that includes the steps above, the expected downtime window, and a verification command (kubectl get --raw='/readyz').
4. Schedule a quarterly drill: manually trigger a Backup CR, then a Restore CR in a test Shoot, and confirm that applications return to normal.