Booting k3OS from USB: Persistent Config and Edge Cluster Deployment

Why USB‑Boot k3OS Matters

When you want a single‑device Kubernetes cluster on a Raspberry Pi, Jetson, or other embedded board, you often face two obstacles: how to load the OS from removable media and how to keep configuration across reboots. k3OS solves both by shipping a read‑only root filesystem and a dedicated data volume that persists between boots. This means you can flash a USB stick once, tweak a tiny config.yaml, and have a fully‑functional k3s node that remembers its network settings, node name, and kubeconfig no matter how many times it’s rebooted.

Core Idea: Read‑Only Root + Data Volume

The k3OS image includes:

• Root partition – a read‑only squashfs that contains the OS and k3s binaries.
• Data partition – a writable block that mounts at /var/lib/k3os during boot.
• Merge logic – the initramfs copies any files found in the data partition into the corresponding locations under the root filesystem, so /var/lib/k3os/config.yaml overrides the default.

Because the root is immutable, you never have to worry about accidental corruption, and updates simply mean reflashing the USB image. All runtime state (etcd snapshots, container images, etc.) lives on the data volume, keeping flash wear low.

Step‑by‑Step: From USB to Running Node

1. Flash the image

   # Replace /dev/sdX with your USB device
   sudo dd if=k3os.img of=/dev/sdX bs=4M status=progress && sync
   

   After flashing, the USB will contain two partitions: k3os-root (squashfs) and k3os-data (ZFS or ext4).

2. Prepare the data volume

   # Mount the data partition on the host
   sudo mkdir /mnt/k3os-data
   sudo mount /dev/sdX2 /mnt/k3os-data
   

   Now edit the config.yaml inside the data partition. Create the file if it doesn’t exist.

3. Edit config.yaml

   # /mnt/k3os-data/config.yaml – minimal example
   node-name: "edge-node-01"
   network:
     interfaces:
       - name: eth0
         dhcp: true
   kubeconfig:
     server: https://10.0.0.1:6443
     token: abcdef123456
   

   Replace the server and token with your cluster’s control plane endpoint and bootstrap token.

4. Unmount and boot

   sudo umount /mnt/k3os-data
   # Insert the USB into the target device and set it as the first boot device in BIOS/UEFI.
   

   On first boot, k3OS mounts the data volume, copies config.yaml into the runtime environment, and starts k3s with the supplied settings.

5. Verify the node

   # From a control plane machine
   kubectl get nodes -o wide
   

   You should see edge-node-01 listed and reporting as Ready.

Why This is a Good Fit for Edge

1. Zero‑touch provisioning – tweak a YAML file and the node comes up with the correct identity and networking.

2. Low storage wear – the root image never changes, so the flash media only sees writes to the data volume.

3. Portable across hardware – the same USB image works on ARM (Raspberry Pi, Jetson) and x86 boards that support UEFI.

4. CI friendliness – spin up a disposable node in a test harness by flashing a fresh USB and feeding it a test config.yaml.

Trade‑offs & Limitations

• Performance – the data volume is often a USB 3.0 or SATA device. On high‑throughput workloads, the I/O can become a bottleneck. Using an NVMe SSD for the data partition mitigates this but requires driver support in k3OS.
• Data volume size – k3s stores container images, etcd data, and logs in /var/lib/rancher/k3s. If the data partition fills, the node will refuse to start services that need persistence. Plan for at least 10 GB for a small cluster.
• Boot order changes – firmware updates on the host can reset the boot priority. Always verify that the USB is first in the boot list after a firmware update.
• No in‑place OS upgrades – updating k3OS requires reflashing the image. If you need frequent OS patches, consider a separate host OS that manages k3s as a container.
• Legacy BIOS machines – k3OS relies on UEFI for the USB boot path. On pure BIOS systems you may need Syslinux or a custom bootloader to load the initramfs.

Practical Checklist Before You Deploy

• Verify the USB device is detected as /dev/sdX and has two partitions.
• Ensure the data partition is formatted with a filesystem that k3OS supports (ZFS or ext4). ZFS gives snapshots but may need extra RAM on small devices.
• Confirm the config.yaml contains a valid kubeconfig token that matches the control plane.
• Set the USB as the first boot device in BIOS/UEFI and disable secure boot if it interferes.
• After boot, run k3s kubectl get nodes to see the node status.
• Check /var/lib/k3os/config.yaml on the node to verify that the merged configuration matches what you set.

Takeaway

k3OS’s USB boot with a persistent data volume turns a single USB stick into a fully self‑contained, zero‑configuration k3s node. By editing a tiny config.yaml before the first boot, you can deploy lightweight clusters on embedded hardware quickly, reliably, and with minimal maintenance. Just remember to size the data partition appropriately and keep an eye on I/O performance if you plan to run container workloads that generate a lot of storage traffic.