Implementing Disaster Recovery with the Rancher Backup Operator

The Problem: Recovering Cluster State After Catastrophic Failure

When a Kubernetes management cluster fails, losing the etcd database means losing all cluster configurations, user permissions, and project settings. Relying on manual snapshots is error-prone and often fails during high-pressure recovery scenarios. The Rancher Backup Operator automates the snapshotting of the cluster state to an external S3-compatible store, providing a verifiable path to restoration.

Prerequisites

• A running Rancher management cluster (v2.6+ recommended).
• An S3-compatible storage backend (e.g., AWS S3, MinIO, or Google Cloud Storage).
• Administrative access to the cluster via kubectl and the Rancher UI.
• A dedicated S3 bucket with read/write permissions for the backup service account.

Configuring the Backup Storage Backend

The Backup Operator does not store data locally; it requires a BackupConfig resource to define where the snapshots are sent. This ensures that even if the entire cluster is deleted, the state remains safe in external storage.

Run the following command on the management cluster as a cluster-admin to create the storage configuration. Replace the placeholders with your specific S3 details:

cat <<EOF | kubectl apply -f -
apiVersion: backup.cattle.io/v1
kind: BackupConfig
metadata:
  name: s3-backup-config
  namespace: cattle-backup-system
spec:
  s3:
    bucket: rancher-backups-bucket
    endpoint: s3.amazonaws.com
    region: us-east-1
    secretRef: s3-credentials
EOF

Note: The s3-credentials secret must exist in the cattle-backup-system namespace and contain the access-key and secret-key keys. If the secret is missing, the operator will enter a CrashLoopBackOff state.

Executing and Scheduling Backups

Once the BackupConfig is applied, you can trigger a manual snapshot to verify connectivity. This is done by creating a Backup resource.

To trigger a manual backup via the CLI:

kubectl apply -f - <<EOF
apiVersion: backup.cattle.io/v1
kind: Backup
metadata:
  name: manual-snapshot-$(date +%Y%m%d)
  namespace: cattle-backup-system
spec:
  backupConfigRef: s3-backup-config
EOF

For ongoing protection, use the Rancher UI to set a schedule: Navigate to Cluster > Backup and define a cron-style schedule (e.g., every 24 hours). This ensures the Backup resources are created automatically by the operator.

Verifying Backup Integrity

A successful backup is not guaranteed by the absence of an error message. Perform these three checks to validate the state:

1. Resource Status: Run kubectl get backup -n cattle-backup-system. The status must be Completed.
2. Storage Check: Log into your S3 console and verify that a new folder matching the backup name has been created and contains the etcd snapshot files.
3. Operator Logs: If the status is Failed, check the operator logs for permission errors:
   kubectl logs -n cattle-backup-system -l app=rancher-backup

Restoring the Cluster State

Restoration is a destructive operation. It reverts the cluster state to the exact moment the snapshot was taken, overwriting any changes made since then.

To restore, create a Restore resource referencing the specific backup name:

cat <<EOF | kubectl apply -f -
apiVersion: backup.cattle.io/v1
kind: Restore
metadata:
  name: recovery-operation
  namespace: cattle-backup-system
spec:
  backupName: manual-snapshot-20260917
  restoreOption: RestoreAll
EOF

Critical Limitation: Restoring a backup typically requires the cluster to be in a state where the Backup Operator is running but the management plane is not actively modifying resources. In cases of total cluster loss, you must first reinstall the Rancher management cluster and the Backup Operator before applying the Restore resource.

Recovery and Rollback

Because a Restore operation changes the state of the entire etcd database, there is no "undo" button. To mitigate risk:

• Pre-Restore Snapshot: Always trigger a manual backup immediately before attempting a restore. This provides a fallback point if the restoration fails or reverts the cluster to an unusable state.
• Staging Validation: Never perform a first-time restore on a production cluster. Deploy a temporary staging cluster and apply the Restore resource there to verify the backup archive is not corrupted.