Diagnosing RKE2 Kubelet Stalls During Rancher 2.7+ Cluster Upgrades

Recognizable Condition

During a Rancher‑initiated RKE2 cluster upgrade, the upgrade progress stalls at the "updating kubelet" step for one or more nodes. The Rancher UI shows the node status as "Upgrading" with no further change, and the upgrade history logs an error indicating a non‑zero exit code from the kubelet start‑up command.

Cause/Diagnostic Table

Possible Cause	What to Look For
Kubelet binary mismatch	Upgrade payload contains a newer kubelet binary but the node still runs the old version; journalctl shows "exec format error" or "cannot execute binary file".
Incompatible container runtime	containerd or CRI‑O version is too old/ new for the target kubelet; logs contain "failed to load plugin …" or "runtime network not ready".
Systemd reload loop due to resource constraints	systemd status shows "active (exited)" then repeatedly "restarting"; journalctl lines "Failed to start RKE2 server: No space left on device" or "Cannot allocate memory".
Unsupported custom kubelet flags	/etc/rancher/rke2/config.yaml contains flags removed in the target version; kubelet start‑up fails with "unknown flag …".
Network/TLS interruption	Upgrade logs show "tls: handshake timeout" or "context deadline exceeded"; node may be unable to reach the Rancher server or the cluster’s API endpoint.
Etcd data inconsistency	Control plane logs show "etcd request failed" or "snapshot mismatch"; kubectl get nodes shows the node as NotReady despite kubelet appearing to start.

Ordered Checks

1. Gather the upgrade error from Rancher UI
   # In the Rancher UI, navigate to Cluster → Upgrade History → View Logs
   Look for the line that begins with "Error updating kubelet on node " and note the exit code and message.
2. SSH to the affected node
   ssh @
   (Requires sudo or root privileges to read journal and systemd units.)
3. Check the kubelet service logs
   journalctl -u rke2-server -n 200 | grep -i kubelet
   or for agents: journalctl -u rke2-agent -n 200 | grep -i kubelet
   Look for:
   • "exec format error" → binary mismatch.
   • "unknown flag" → unsupported kubelet config.
   • "No space left on device" or "Cannot allocate memory" → resource constraints.
   • runtime‑related errors → containerd/CRI‑O version.
4. Inspect the kubelet binary version
   /var/lib/rancher/rke2/bin/kubelet --version
   Compare with the version declared in the Rancher upgrade payload (visible in the upgrade log as "Target kubelet version: vX.Y.Z").
5. Verify container runtime version
   containerd --version or crio --version
   Check against the RKE2 version compatibility matrix (e.g., RKE2 v1.27 requires containerd ≥1.6).
6. Check systemd unit state and resource usage
   systemctl status rke2-server
df -h /var/lib/rancher/rke2
free -m
7. Review custom kubelet configuration
   cat /etc/rancher/rke2/config.yaml
   Search for flags that are not listed in the target version’s kubelet --help output.
8. Test network connectivity to the API endpoint
   curl -k https://:6443/healthz
   If the request times out or fails with TLS errors, note the network issue.
9. Validate etcd health (only if control plane is accessible)
   etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/rancher/rke2/tls-etcd/ca.crt --cert=/etc/rancher/rke2/tls-etcd/server.crt --key=/etc/rancher/rke2/tls-etcd/server.key endpoint health

Fixes Tied to Findings

• Kubelet binary mismatch
  Re‑run the upgrade step for the node via Rancher UI (Cluster → Nodes → Actions → Upgrade Node). If the UI button is unavailable, manually synchronize the binary:
  # On the node, as root
systemctl stop rke2-server # or rke2-agent
rm -f /var/lib/rancher/rke2/bin/kubelet
cp /var/lib/rancher/rke2/data/*/bin/kubelet /var/lib/rancher/rke2/bin/kubelet
systemctl start rke2-server
• Incompatible container runtime
  Upgrade containerd or CRI‑O to a version supported by the target RKE2 release, then restart the kubelet service:
  # Example for containerd on Ubuntu
sudo apt-get update && sudo apt-get install -y containerd=1.6.*
sudo systemctl restart containerd
sudo systemctl restart rke2-server
• Systemd reload loop due to resource constraints
  Free disk space or memory, then restart the service:
  # Clean old logs and containers
journalctl --vacuum-time=3d
crictl prune
df -h / (ensure >15% free)
  systemctl restart rke2-server
• Unsupported custom kubelet flags
  Edit /etc/rancher/rke2/config.yaml to remove or replace the offending flag, then reload:
  # Example: remove the deprecated --experimental‑mounter flag
sudo sed -i '/--experimental-mounter/d' /etc/rancher/rke2/config.yaml
sudo systemctl daemon-reload
sudo systemctl restart rke2-server
• Network/TLS interruption
  Verify that the node can reach the Rancher server and the cluster API endpoint:
  # Check connectivity
curl -v https://:80
curl -k https://:6443/healthz
  If TLS handshake fails, ensure the node’s clock is synchronized (NTP) and that the CA certificates in /etc/rancher/rke2/tls are up to date.
  sudo systemctl restart rke2-server
• Etcd data inconsistency
  Do **not** manually edit etcd data. Instead, use Rancher’s etcd snapshot restore or replace the node:
  # From a healthy control plane node
rancher etcd snapshot list
rancher etcd snapshot restore --target‑node
  After restore, reboot the node and let Rancher re‑add it to the cluster.

Escalation Criteria

Escalate to Rancher support or the underlying Kubernetes vendor when:

• The node remains in the "updating kubelet" state after completing all checks and applying the corresponding fix.
• Multiple nodes exhibit the same error pattern, suggesting a systemic issue (e.g., version incompatibility across the fleet).
• Etcd health checks fail and you cannot restore from a known good snapshot.
• You encounter panic messages or kernel oops in journalctl that point to a deeper OS or hardware problem.