k3OS Self‑Upgrading OS: Immutable Infrastructure Made Practical

Why an OS‑as‑a‑Resource Matters

Modern Kubernetes clusters often treat nodes as disposable, but that approach hides a subtle source of drift: the operating system itself. When a node’s kernel or userspace packages get out of sync with the control plane, security audits become harder and upgrade windows widen. k3OS solves this by turning the OS into a Kubernetes custom resource (CRD) that the cluster can manage itself. The result is a fully immutable, cluster‑driven upgrade path that eliminates manual OS re‑image steps.

How k3OS Models the OS

k3OS exposes a k3os.cattle.io/v1.OS CRD. A node’s OS version is stored in the spec.version field. Updating that field triggers the built‑in k3os-upgrade DaemonSet, which runs on every node. The DaemonSet’s pod image contains k3os-install, a lightweight tool that writes the new OS image into a dedicated partition and updates the bootloader.

Because the root filesystem is mounted read‑only and only writable overlays exist at /etc and /var, the OS image itself never changes in place. This architecture supports true immutable infrastructure: the same image lives on every node until the cluster explicitly upgrades it.

Two‑Phase Upgrade Flow

1. Stage: The DaemonSet downloads the target image (e.g., rancher/k3os:v0.24.4) and writes it to the /dev/sda2 partition. It also updates the GRUB or UEFI boot entry to point to this new partition.
2. Reboot & Verify: The node reboots into the new image. A health probe checks that k3s and the OS itself are reachable. If the probe fails, the bootloader automatically falls back to the previous partition, allowing a zero‑downtime rollback.

Because the upgrade logic lives inside the cluster, you can trigger it from any node that can run kubectl, even if the node you intend to upgrade is offline.

Concrete Upgrade Example

Assume a cluster already running k3OS v0.24.3. To upgrade to v0.24.4:

# Run on any control‑plane node
kubectl apply -f - <<'EOF'
apiVersion: k3os.cattle.io/v1
kind: OS
metadata:
  name: node-01
spec:
  version: rancher/k3os:v0.24.4
EOF

Replace node-01 with the name of the target node. The DaemonSet will pick up the change, stage the image, and reboot the node. You can watch the progress with:

kubectl get nodes -o wide
kubectl logs -n kube-system -l app=k3os-upgrade -c upgrade

After the reboot, verify that the node reports the new version:

kubectl get os node-01 -o jsonpath='{.status.version}'

Risk: If the new image is corrupted, the node will reboot into the previous partition automatically. However, if the bootloader entry fails to update (e.g., due to a firmware bug), you may need console access to correct the boot entry manually.

Trade‑offs and Limitations

• Unmaintained Core: k3OS is archived as of v0.24.4. The underlying kernel (5.10 LTS) and BusyBox userspace are no longer patched. Production use requires accepting this risk.
• No Package Manager: The minimal ISO purposely omits apt or apk. If you need custom binaries, you must build a custom image or use a sidecar container, breaking the immutable model.
• Cloud‑Init Dependence: Initial boot is driven by cloud‑init. Bare‑metal PXE installs need custom scripts; unsupported datasources may fail to provision.
• Control‑Plane Downtime: During a control‑plane node upgrade, the k3s server pod restarts, briefly dropping API access. For high‑availability, upgrade other nodes first.

How to Verify the Result

After an upgrade, confirm the read‑only root by:

mount | grep /dev/root

Overlay mounts at /etc and /var should appear:

findmnt -t overlay

These checks ensure that the OS image remains immutable and that only the writable layers contain state.

Actionable Takeaway

If your cluster requires a reproducible, auditable OS layer, k3OS’s CRD‑driven upgrade model is a powerful tool. Just remember that the project is unmaintained, so plan for custom images or alternative tooling if you need long‑term security support. The upgrade workflow is simple: update the OS CRD, let the DaemonSet roll it out, and let the bootloader handle rollback if something goes wrong.