Running Kubernetes on k3OS: Immutable Root and OverlayFS Updates

Configuration drift kills remote edge nodes

You receive an alert that a Kubernetes node at a wind farm or retail site has lost DNS resolution. You log in over a spotty cellular link, edit /etc/resolv.conf, restart kubelet, and the workloads recover. A week later you reboot the device for a power maintenance window, and the DNS fix is gone. Worse, five other gateways never got the edit at all, so your fleet is silently inconsistent.

This is the problem k3OS is built to prevent. Its root filesystem is intentionally immutable: system binaries live on a read-only partition, and updates replace the boot image rather than patching files in place. For edge Kubernetes clusters, that means every node boots from a known-good OS state, but it also changes how you troubleshoot and deploy.

Read-only root, writable overlay

k3OS mounts the root partition read-only and stacks a writable layer on top using overlayfs, a union filesystem that presents a single merged view to applications. Writes to /var/lib/kubelet, logs, and temporary update artifacts go into the overlay, while the underlying OS image stays unchanged. Kubernetes components such as kubelet and containerd run normally because their executables are on the immutable layer and the overlay supplies the writable directories they expect.

Because the base image never changes, there is no apt upgrade workflow that gradually drifts one node away from another. The OS is deployed as an image, not maintained as a collection of edited files.

How updates are applied

An update in k3OS creates a new kernel and initrd pair, installs them alongside the previous pair, and updates the boot entry so the next reboot starts from the new image. The previous system remains on disk untouched. If the new image fails to boot, the operator can select the earlier boot entry at the bootloader and return to the known-good state.

This is an image-level replacement, not an in-place patch. It is well suited to remote devices where a failed package upgrade can leave a node unreachable, but it means you cannot fix a broken configuration by editing /etc and expecting the change to survive a reboot.

Verify the immutable layout before updating

Run these checks on the k3os node, either at the local console or over SSH. You need root or sudo privileges to inspect mount details and run the update tool.

1. Confirm the OS identity and read-only flag:

grep -E 'K3OS|ReadOnlyRootFilesystem' /etc/os-release

Expected output includes lines similar to K3OS="k3os" and ReadOnlyRootFilesystem=true. Exact formatting can vary by build.

1. Inspect the active root mount:

mount | grep ' on / '

You should see the root device mounted ro and an overlay mount marked rw. If the root shows rw, the system may not be in its default immutable state.

1. Apply an update and reboot:

sudo k3os update
sudo reboot

After the node comes back, run the mount check again. The overlay should reference a newer kernel/initrd version than before. If it does not, the update did not become active.

Risk: Losing power during the update window can leave the new boot entry partially written. Because the previous image is preserved, you can usually fall back at the bootloader, but a remote site without physical access may need a recovery image.

What breaks on an immutable root

Any script or installer that assumes it can write to /etc, /usr, or other system directories will fail unless it targets the writable overlay or a persistent data partition. On very low-end hardware, overlayfs adds a small CPU and memory cost for every file operation, which can affect dense container workloads.

Practical check: after any change, reboot the node and confirm the change persists. If it disappears, the change was written to the overlay and discarded or never applied to the base image.

Operate k3OS like an appliance

Treat each k3OS node as an appliance image rather than a general-purpose Linux server. Build and test the OS image with the configuration you need, deploy it to the fleet, and validate with the mount and os-release checks above. Reserve SSH for diagnostics, not for permanent configuration edits.