k3OS Rolling Updates: How Mirantis' Built‑in Upgrade Service Keeps Your Cluster Fresh

Why a Rolling Update Matters in a Tiny Kubernetes Distribution

Running Kubernetes on bare metal or virtual machines is a common way to keep a production cluster lean. In that scenario, the k3OS operating system bundles a minimal Alpine Linux base with systemd and containerd, then runs the k3s Kubernetes binary as the primary service. Because k3OS is designed to be a single‑node or multi‑node cluster, the vendor‑provided update workflow is a key part of keeping the cluster secure and feature‑rich without a full OS reinstall.

What the Mirantis Update Service Actually Does

Mirantis ships k3OS with an optional k3s‑upgrade command that talks to a signed container image registry. The update service follows a declarative release channel (e.g., stable, edge) and pulls the latest signed k3s binary as a container layer. Because the binary is delivered inside a container image, the upgrade is essentially a layer replacement rather than a full filesystem diff, which keeps network traffic low and makes rolling back to a previous release trivial.

When you run k3s-upgrade --channel stable on a node, the tool:

• downloads the newest k3s image from the Mirantis registry;
• replaces the local binary with the new one;
• restarts the k3s service;
• reports the new k3s_version via systemctl status k3s and kubectl get nodes.

The upgrade runs sequentially across nodes, respecting any node taints that mark a node as unschedulable. This means the cluster continues to accept traffic on the remaining nodes while a single node is being updated.

Practical Example: Upgrading a 3‑Node k3OS Cluster

# On the first node
sudo k3s-upgrade --channel stable
# Verify the upgrade locally
sudo systemctl status k3s | grep k3s_version
# On the cluster
kubectl get nodes -o wide
# Repeat on the second and third nodes

After each node restarts, kubectl get nodes should show the node as Ready without any pending pods. The upgrade log lives in /var/log/k3s/k3s-upgrade.log and can be grepped for "error" to confirm a clean run.

Key Verification Steps

1. Check the node’s k3s version before and after the upgrade:

   kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}'
   

2. Ensure etcd remains healthy:

   kubectl get pods -n kube-system -l component=etcd -o wide
   

3. Confirm the upgrade log has no error entries:

   grep -i error /var/log/k3s/k3s-upgrade.log
   

Trade‑offs and Limitations

While the built‑in upgrade service is convenient, it comes with a few caveats:

• Subscription Requirement: The signed image registry is only available to Mirantis customers. Community‑built k3OS images must perform manual binary replacement or use third‑party scripts.
• etcd Health: The upgrade process assumes a healthy etcd cluster. A split‑brain or unresponsive etcd node can cause the upgrade to fail or leave the cluster in an inconsistent state.
• Version Restriction: The automated upgrade path is officially supported only on k3OS 1.25+; older releases may need manual intervention.

When to Use the Built‑in Upgrade vs. Manual Replacement

If you are a Mirantis customer running a production cluster, the k3s-upgrade command is the safest and most auditable path. It guarantees signed binaries, respects node taints, and provides a straightforward rollback via the previous release tag. For community users or environments where a subscription is not feasible, the manual route—downloading the k3s binary from the official release page and replacing /usr/local/bin/k3s—remains viable but loses the declarative channel benefits.

Actionable Takeaway

To keep your k3OS cluster up to date with minimal disruption:

• Subscribe to Mirantis’ update service or ensure your image registry is reachable.
• Run k3s-upgrade --channel stable on each node, one at a time.
• Validate the node’s readiness and etcd health after each upgrade.
• Keep the /var/log/k3s/k3s-upgrade.log handy for audit and rollback.

By following this cadence, you can maintain a secure, feature‑rich Kubernetes cluster without the overhead of full OS re‑deployments.