k3OS Immutable Root and Image Updates for Reliable Edge Nodes

When a reboot erases your fixes, the node was never meant to be fixed in place

Edge k3s clusters tend to accumulate manual changes. A quick SSH to install a diagnostic binary, a tweak to /etc, a package added to work around a driver. After a power loss the node comes back clean and the change is gone, or worse the change persists on one node and not the others. k3OS is built around the engineering decision to make that class of drift impossible by treating the operating system as an immutable image with a read-only root and a writable overlay for state.

The practical takeaway is simple: system files do not change on a running k3OS node. Configuration and runtime state live in designated writable areas, and updates replace the whole OS image atomically. Nodes are intended to be disposable and identical, with customization done at image build time or via Kubernetes, not via ad-hoc shell access.

Read-only root with a writable overlay

k3OS ships as a minimal Linux distribution where k3s is the intended workload. The root filesystem is mounted read-only. Stateful paths such as /var and /etc are presented through a writable overlay so Kubernetes state, logs and runtime data can persist while system files remain immutable.

This split reduces attack surface and size by removing a general package manager and multi-user tooling. It also enforces a clear boundary: writes to system paths are not durable across reboots unless they target the overlay-backed locations.

How to verify the layout on a node

Run on the node as root. Inspect mount options for the root and state paths.

mount | grep ' / '

Check for read-only options on /. Then inspect the state mounts.

findmnt -o TARGET,SOURCE,OPTIONS /var

Expected checks: root shows ro, and /var and /etc are backed by an overlay or separate writable mount. Risk: attempting to write to a read-only system path will fail, and changes made outside the overlay are lost on reboot.

Declarative first-boot provisioning

Node provisioning is declarative at first boot using cloud-init style inputs. The same image can be reused across nodes without post-install package changes. k3s configuration, cluster join details and cloud metadata are supplied as inputs rather than applied interactively.

Because there is no package manager, ad-hoc debugging tools must be baked into a custom image or provided via containers. This is intentional. System-level customization is expected to happen before the image is deployed, not after.

Image-based atomic updates

Updates are image-based rather than incremental package upgrades. A new OS image is downloaded and activated atomically, with rollback support. This aligns with immutable infrastructure principles and avoids partial upgrade states on constrained edge hardware.

Image updates require reliable network access to fetch the new image and enough storage for the new root. On storage-constrained devices this can be a limiting factor. Behavior is version-sensitive; overlay layout, provisioning keys and the update flow can change between releases, and project maintenance status should be reviewed for long term support.

Trade-off to plan for

The minimal design is a feature until you need an interactive fix. Operators accustomed to mutable servers are surprised when changes to system paths disappear after reboot. Debugging must be planned ahead: include needed tools in the image, use ephemeral containers for diagnostics, and keep configuration in Kubernetes or the first-boot inputs.

Limitations to keep in mind: no traditional package manager, read-only root surprises, network and storage requirements for image updates, and version sensitivity of overlay and provisioning mechanisms.

Actionable check before production

Boot a test image in a VM or lab node and confirm the mount options for / and /var. Observe first-boot behavior with declarative inputs to verify provisioning occurs without package manager use. Examine the update workflow to confirm it downloads a full OS image and activates it atomically rather than applying package patches. Review release notes and project status indicators to assess current maintenance and supported k3s and kernel versions.