k3os-node-agent configuration persistence limits

Answer

The k3os-node-agent treats any file that lives on the writable state partition (typically labeled K3OS_STATE or mounted at /oem) as persistent configuration. Everything else that resides on the read‑only root‑filesystem overlay is considered transient and is discarded when the immutable image is replaced on reboot.

How the agent distinguishes transient vs. persistent state

• During boot the agent mounts the state partition under /var/lib/rancher/k3os (or /oem depending on the image).
• It reads /etc/rancher/k3os/config.yaml from that location. If the file is present there, changes survive a reboot because the partition is not recreated from the immutable image.
• If the same path is edited on the overlay (the default read‑only rootfs), the overlay is rebuilt from the base image each boot, so the edits are lost.

Documented mechanisms for injecting persistent node‑specific metadata

1. Place custom configuration (e.g., config.yaml, scripts, or certificates) directly on the state partition:
• Mount the state partition (often /dev/disk/by-label/K3OS_STATE) at /mnt/state.
• Copy your file to /mnt/state/etc/rancher/k3os/config.yaml or /mnt/state/oem/config.yaml.
• Unmount and reboot; the agent will read the persisted file.
2. Use the /oem directory, which is typically a symlink to the state partition and is writable:
• Edit /oem/config.yaml or add files under /oem.
• These survive image swaps because /oem is backed by the state partition.
3. Leverage cloud‑init user‑data to inject configuration at first boot:
• Provide a write_files section that targets /oem or /var/lib/rancher/k3os.
• Cloud‑init runs before the agent starts, ensuring the file is on the persistent partition.

Verification steps for this case

1. Identify the state partition mount point:

# Find the device labelled K3OS_STATE
lsblk -f | grep K3OS_STATE
# Or check /etc/fstab for the /oem mount
grep oem /etc/fstab

2. Check available space on that partition:

df -h /mnt/state   # replace with your mount point

3. Confirm your custom file resides on that mount:

mount | grep /etc/rancher/k3os/config.yaml
# Should show a device backed by K3OS_STATE or /oem

4. After a reboot, verify the file content is unchanged:

cat /etc/rancher/k3os/config.yaml

Note: If the state partition is nearly full, large config blobs may be truncated or cause boot failures. The practical limit is the free space reported by df -h on the state partition.