Declarative Node Configuration in k3OS: Using k3os.yaml with an Immutable Root Filesystem

Manually configuring Kubernetes nodes with SSH, ad-hoc package installs, and hand-edited service files creates drift: two nodes that should be identical slowly diverge. k3OS addresses this by pairing an immutable root filesystem with a single declarative YAML file. The useful takeaway is that any state you want to survive a reboot must live in /etc/k3os/k3os.yaml; everything else is disposable.

Why k3OS Puts Node State in One YAML File

k3OS is a specialized Linux distribution that embeds the k3s binary directly into the OS image. Instead of installing packages and editing configuration files on a running node, you define the desired state once in a YAML file. At boot, the system reads that file and applies hostname, SSH keys, network settings, and k3s server or agent flags. Because the root filesystem is immutable, the node behaves like an appliance: you replace the whole image to update, rather than patching individual packages.

The Boot-Time Configuration Mechanism

The primary configuration file is /etc/k3os/k3os.yaml. During the boot process, k3OS parses this file before starting k3s. The file is declarative: you describe the end state, and the boot sequence reconciles the running system to match it. This is the same operational model used by cluster provisioning tools, but applied to the operating system itself.

What the Immutable Root Filesystem Means

An immutable root filesystem means the OS partition is read-only or reset on every boot. If you install a package, edit a file under /etc, or drop a script into /usr/local/bin, that change is discarded when the machine restarts. Persistent state belongs in one of three places: the k3os.yaml file, a mounted data volume, or the Kubernetes API itself. This constraint eliminates drift but requires a mindset shift if you are used to general-purpose distributions such as Ubuntu or Fedora.

Worked Example: A Server Node k3os.yaml

The following block illustrates the structure of a server-node configuration. It is an example only; key names and supported fields vary by k3OS release, so verify the schema against the documentation bundled with your image before deploying.

# /etc/k3os/k3os.yaml
# Illustrative structure — confirm keys against your k3OS version.
hostname: edge-server-01
ssh_authorized_keys:
  - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... operator@workstation
k3s_args:
  - --disable=traefik
  - --disable=servicelb
  - --write-kubeconfig-mode=644
  - --node-ip=10.0.4.15

In this example, the hostname field sets the node name, ssh_authorized_keys installs a login key without manual user creation, and the k3s_args list passes flags directly to the embedded k3s server process. Disabling Traefik and ServiceLB is a common choice when you plan to run your own ingress controller or load balancer.

Applying and Checking the Configuration

To apply the file, place it at /etc/k3os/k3os.yaml on the node's persistent storage or supply it through the boot mechanism your installation uses. The following commands assume you are already logged in as root or have sudo access.

# Run on the k3OS node as root or via sudo
sudo mkdir -p /etc/k3os
sudo cp /path/to/k3os.yaml /etc/k3os/k3os.yaml
sudo reboot

Because this operation changes node state, plan a rollback path: keep a known-good copy of the previous k3os.yaml. If the new configuration fails, replace the file with the backup and reboot again.

After the node returns, verify that the boot process consumed the file and that k3s started with the expected arguments.

• Check the hostname: hostname should return edge-server-01.
• Inspect the k3s process: ps aux | grep k3s should show the flags from k3os.yaml.
• Read the boot logs: sudo journalctl -u k3s (or sudo journalctl | grep k3os) should show the configuration being parsed and applied.
• If this is a server node, confirm cluster membership: sudo k3s kubectl get nodes.

To confirm the immutable behavior, attempt a manual change to a system directory, reboot, and check whether it persists.

# Run on the k3OS node as root or via sudo
sudo touch /opt/k3os-immutable-test
sudo reboot
# After reboot:
ls /opt/k3os-immutable-test

If the file is absent after reboot, the root filesystem reset as expected. If the write itself fails because the directory is read-only, that also confirms the immutable design.

Limits, Failure Modes, and Common Mistakes

The declarative model removes drift, but it introduces its own failure modes.

Mistake or Limit	Consequence	Mitigation
Editing files directly on the node	Changes are lost on reboot	Move all persistent settings into k3os.yaml or a data volume
YAML syntax error in k3os.yaml	Boot may fail or the node may not join the cluster	Validate YAML offline before copying it to the node
Expecting a general-purpose package manager	Tools such as apt or dnf are unavailable	Use only the binaries included in the image, or build a custom image
Assuming active upstream maintenance	Security patches or k3s upgrades may lag	Verify the project's current status and your image's support window before production use

A malformed k3os.yaml is particularly dangerous because it can create a boot loop: the system restarts, fails to parse the configuration, and restarts again. If a node becomes unreachable after a config change, boot from installation media, mount the persistent partition, and repair or remove the offending file.

When This Pattern Fits

k3OS suits edge deployments, CI runners, and disposable cluster nodes where you want every machine to be identical and replaceable. It is a poor fit when you need persistent local state outside of Kubernetes, frequent interactive debugging with arbitrary tools, or a long-term general-purpose server. Before committing, confirm that the k3OS release you plan to use still matches your k3s version requirements and operational support expectations.