Safely Upgrading Kubernetes Clusters at Scale with Rancher’s Built‑in Upgrade Workflow

Managing dozens of Kubernetes clusters often turns a routine version bump into a high‑risk project. Teams must coordinate node drains, workload rescheduling, etcd backups, and rollback procedures while keeping an audit trail. Rancher’s upgrade workflow addresses this by turning a scattered set of manual steps into a single, version‑controlled operation that can be triggered from the UI or API.

How the workflow works

When you select a cluster in Rancher and choose Upgrade Kubernetes, the server reads the cluster’s Cluster custom resource, compares the desired spec.kubernetesVersion with the current status, and triggers the embedded cluster controller (for RKE2, K3s, or imported clusters). The controller watches for version changes, runs pre‑upgrade validation checks, and orchestrates the upgrade node‑by‑node while respecting PodDisruptionBudgets. If any step fails, the controller automatically rolls back to the previous Kubernetes version and raises an alert.

Pre‑upgrade checklist

Before the upgrade begins, Rancher presents a checklist that typically includes:

• etcd snapshot of the target cluster
• Node drain safety (respecting PDBs)
• Image compatibility verification for the target version
• Confirmation that the chosen Kubernetes version is supported by the node OS and any custom operators

Only after you confirm each item does Rancher proceed.

Worked example: upgrading an RKE2 cluster to v1.28.2

Assume you have Rancher 2.7+ installed and you manage an RKE2 cluster named prod‑west. You want to move from Kubernetes v1.27.3 to v1.28.2.

1. Access the UI: Log in to Rancher with a user that has the cluster-admin role on the target cluster.
2. Navigate: Global → Clusters → prod‑west → Edit → Upgrade Kubernetes.
3. Select version: In the dropdown, choose v1.28.2 (RKE2). Rancher will display the pre‑upgrade checklist mentioned above.
4. Run validation: Click Run Checks. Rancher will verify etcd backup availability, node drain feasibility, and image compatibility. If any check fails, resolve the issue before proceeding.
5. Start upgrade: After all checks pass, click Upgrade. Rancher begins upgrading control‑plane nodes one‑by‑one, then worker nodes, updating the Cluster CR’s spec.kubernetesVersion as it goes.
6. Monitor progress: The cluster detail page shows a progress bar and logs each node’s status. You can also watch the upgrade from the command line:

# Run on a workstation with kubectl configured for the cluster
# Requires read‑only access to the cluster; no special permissions needed.
kubectl get nodes -o wide
# Look for the KUBELET VERSION column; it will change from v1.27.3 to v1.28.2 as nodes finish.

If a node fails to upgrade (e.g., due to a kubelet incompatibility), Rancher automatically:

• Drains the node back to the previous version
• Restores the etcd snapshot taken before the upgrade
• Updates the Cluster CR to reflect the original Kubernetes version
• Logs an action: cluster.upgrade.failed entry in the audit log

Trade‑off and limitation

The upgrade orchestrator runs inside the Rancher server’s embedded etcd. When many clusters are upgraded concurrently, the controller’s loop can experience increased latency, which may lengthen the total upgrade window. Additionally, clusters that use a custom CNI or rely on non‑standard node labels may need manual reconciliation after the upgrade, because the controller only resets labels that it manages.

To mitigate latency, consider:

• Staggering upgrades across maintenance windows
• Monitoring the Rancher server’s etcd request latency via the rancher-etcd-requests metric (if you have Prometheus scraping enabled)
• Limiting the number of simultaneous upgrades via the Global → Settings → Upgrade → Max concurrent upgrades setting

Actionable closing

Before rolling out upgrades to production:

1. Enable the upgrade feature under Global → Settings → Upgrade (ensure it is turned on).
2. Test the entire process on a staging cluster that mirrors your production node OS and operator versions.
3. Verify success with two checks:
• kubectl get nodes – the KUBELET VERSION column should show the target version for all nodes.
• Rancher audit log – look for entries with action: cluster.upgrade and result: success (located at /var/log/audit.log inside the Rancher container).

Once validated, schedule production upgrades during a maintenance window, monitor cluster health through Rancher’s monitoring integration, and keep the etcd snapshot of the Rancher server handy as a safety net.