k3os config partition: persisting /var/lib development state across reboots without manual disk orchestration

Question

Repeatable dev environment on k3os
I am setting up a repeatable development environment on k3os (v0.21.x-era releases, k3s v1.21+ assumed) and want host-level state to survive reboots the same way the OS configuration does. My understanding is that k3os keeps its declarative config on a dedicated partition mounted at /k3os/config, so kernel parameters, the packages: list, and runtime settings persist across rebuilds while the root filesystem stays read-only.
Where the model breaks down
The unresolved part for me is stateful data. Changes under /var/lib or /etc appear to be lost on reboot unless a separate block device is mounted via the disks: section of the config. That makes a single-node dev box awkward: I would like container images, local volumes, and a few host-side tools to persist without hand-managing extra disks on every machine I provision.
Questions

Is mounting a dedicated device at /var/lib (or a subpath) through the disks: section the only supported way to keep stateful development data across reboots, or does the config partition itself allow any writable host paths?
If a packages: entry diverges from k3s default data directories, are there documented overlayFS precedence conflicts I should verify before relying on this in a repeatable setup?

Tasadduq Burney · Answer

Short answer
A standard disk install (k3os install /dev/sda) creates a persistent partition (labeled K3OS_STATE) that automatically mounts at /var/lib/rancher, /var/lib/kubelet, and /var/lib/containerd. Container images, local volumes, and k3s state survive reboots with no disks: configuration required. The /k3os/config partition is read‑only and holds only the declarative YAML; it does not provide writable space for /var/lib or /etc.

When you do need disks:

Running k3os in live/ISO/PXE mode (no persistent partition) — all of /var/lib is tmpfs and lost on reboot.
You need durable host paths outside /var/lib/rancher (e.g., /opt/tools, /home/user).

OverlayFS and packages: — no precedence conflicts
The packages: list installs Alpine packages into the read‑only rootfs overlay. Runtime writes from those packages follow the same persistence rules: they persist only if the target directory lives on the persistent partition. There are no documented overlayFS precedence conflicts between packages: and the default state partition.

Verification steps (run once on a test VM)

k3os install /dev/sda (or your target disk).
lsblk -f — confirm a second partition labeled K3OS_STATE mounted at /var/lib/rancher.
echo test > /var/lib/rancher/k3s/testfile, reboot, then verify the file remains.
cat /proc/mounts — you should see overlayfs on / and a separate ext4/xfs mount on /var/lib/rancher.

One diagnostic detail that changes the recommendation
Are you installing to disk (k3os install /dev/sda) or running in live/ISO/PXE mode? If live mode, you must add a disks: entry (block device or network share) to get any persistent /var/lib state.

Caveat
k3os is end‑of‑life (archived by Rancher/SUSE since 2022). For long‑lived dev boxes consider k3s on Talos, Flatcar, or Ubuntu where persistence is more flexible and security updates continue.

k3os config partition: persisting /var/lib development state across reboots without manual disk orchestration

Repeatable dev environment on k3os

Where the model breaks down

Questions

1 answer

Short answer

When you do need `disks:`

OverlayFS and `packages:` — no precedence conflicts

Verification steps (run once on a test VM)

One diagnostic detail that changes the recommendation

Caveat

Your perspective belongs here.

0 question comments