Running Immutable Edge Nodes with k3OS: When and How to Leverage Its Read‑Only Root

The problem: keeping edge Kubernetes nodes stable and updatable

Edge locations often run limited‑hardware Kubernetes clusters that must stay online for weeks or months. Manual package updates or ad‑hoc configuration drift can break the node, and rolling back a bad change is painful when you have to SSH into dozens of devices.

Thesis: k3OS’s immutable, read‑only root filesystem gives you atomic upgrades and a predictable base, but you must shift workloads and customizations into containers or the image‑build process.

How the immutability works

When k3OS boots, it loads a squashfs image that is mounted read‑only on /. A thin overlayfs layer provides writable storage for /var/lib and /etc only. Because the base cannot be altered at runtime, the OS version is fixed until the node reboots onto a new squashfs image.

Why this matters for edge Kubernetes

• Atomic upgrades: Replacing the image and rebooting swaps the entire OS in one step; there is no intermediate state where half the packages are updated.
• Rollback simplicity: If the new image fails to boot, the bootloader can fall back to the previous image, instantly restoring the known‑good state.
• Reduced attack surface: No package manager on the host means no accidental installation of unnecessary tools.

Worked example: upgrading a fleet of k3OS nodes

Assume you have a set of edge nodes booting from an image hosted at http://repo.example.com/k3os-v1.2.0.iso. You have built a newer image k3os-v1.3.0.iso that includes a security patch and a updated k3s version.

1. Place the new image where nodes can reach it. On your image server, copy the ISO to the same directory and update the symlink or HTTP redirect that points latest-k3os.iso to the new file.
2. Trigger a reboot. From a management host, run:

# Replace  with a space‑separated list of hostnames or IPs
for node in ; do
  ssh root@$node "reboot"
done

Each node will:

• Fetch the ISO via its bootloader (iPXE, GRUB, or similar) on next boot.
• Mount the new squashfs as the read‑only base.
• Bring up the overlay for /var/lib and /etc.
• Start the k3s system service automatically.

To verify the upgrade succeeded on a node, run (as root or with sudo):

# Check the overlay mount
mount | grep 'overlay on /' && echo "Overlay active – base is read‑only"
# Confirm OS version
. /etc/os-release && echo "ID=$ID VERSION_ID=$VERSION_ID"
# Ensure k3s is running
systemctl is-active k3s && echo "k3s is active" || echo "k3s failed"

If any node fails to boot, the bootloader can be configured to try the previous image (many edge bootloaders support a fallback entry). No manual package cleanup is required.

Trade‑off and limitation: host‑level customizations

Because the root filesystem is immutable, you cannot apt install a driver or add a system package after deployment. If you need a custom kernel module, you must:

1. Include the module in the squashfs during image build (using the k3os build scripts or a custom Dockerfile that outputs an ISO).
2. Reboot the node onto the new image.

This shifts the responsibility for host‑level changes to the image‑build pipeline, which is a benefit for reproducibility but requires a build environment and version‑controlled image artifacts.

Practical way to check the result

After a node boots, confirm that writes only go to the overlay:

# Create a test file in a writable path
touch /var/lib/k3os-test && echo "writable layer works"
# Attempt to write to a protected path (should fail)
touch /usr/local/test 2>&1 | grep -i "permission" && echo "Base is read‑only as expected"

If the second command reports a permission error, the immutable base is intact.

Actionable closing

For edge Kubernetes clusters where stability and predictable upgrades outweigh the need for ad‑hoc host packages, k3OS offers a clear path: build a new ISO, roll it out via a simple reboot, and verify the overlay and k3s service. Keep host‑level customizations to a minimum—prefer containerized workloads or include them in the image build. By treating the OS as an immutable artifact, you gain atomic upgrades, easy rollback, and a smaller attack surface, all of which are essential for reliable edge operations.