What k3OS Got Right About Immutable Infrastructure: A Look Back at config.yaml

k3OS is dead — Rancher archived the project around 2022 — but its central idea is still worth studying if you build edge or lab clusters today. The idea: the entire operating system, from hostname to Kubernetes cluster membership, is declared in one file applied at boot. No SSH-ing in to fix things. No configuration drift. If a machine misbehaves, you replace it, because rebuilding it is a config file away.

This post looks at that design decision — the config.yaml model — what it enabled, what it cost, and how to apply the same thinking with tooling that is still maintained.

One file, whole machine

k3OS was a minimal Linux distribution built to run exactly one thing: k3s, Rancher's lightweight Kubernetes. It shipped no package manager and almost no userland tooling. Instead, nearly everything about the system lived in a cloud-init-style YAML file, typically at /var/lib/rancher/k3os/config.yaml (or supplied via a remote URL at boot):

• hostname and ssh_authorized_keys for access
• network configuration (interfaces, DNS, Wi-Fi on supported hardware)
• k3s_args — the arguments passed to the k3s server or agent process
• the cluster token and server URL used for multi-node bring-up

That last point is the interesting one. Because the cluster join information was part of the OS config, a fleet of identical machines could bootstrap themselves: the first node came up as a server, and every additional node read the same token and URL and joined as an agent on first boot. No kubeadm join, no manual step per machine.

A worked example: a self-assembling three-node cluster

Here is the shape of the configuration, based on the documented schema from the archived rancher/k3os repository. Check the release notes for the specific version you use, since the schema evolved between releases.

Server node (config.yaml):

hostname: k3os-server-1
ssh_authorized_keys:
  - ssh-ed25519 AAAA... you@workstation
k3os:
  token: "CHANGE-ME-long-random-string"
  k3s_args:
    - server
    - "--disable=traefik"

Agent nodes:

hostname: k3os-agent-1
ssh_authorized_keys:
  - ssh-ed25519 AAAA... you@workstation
k3os:
  server_url: "https://192.0.2.10:6443"
  token: "CHANGE-ME-long-random-string"
  k3s_args:
    - agent

Where to apply it: write the file onto the install media or serve it over HTTP and point the kernel boot parameter at it. You need console or out-of-band access to the machine for the first boot; after that, SSH with the authorized key is the intended path.

To verify the result, from the server node (or via kubectl with the kubeconfig from /etc/rancher/k3s/k3s.yaml):

kubectl get nodes -o wide

You expect three entries in Ready state. If an agent never appears, the usual suspects are a wrong server_url, a token mismatch, or port 6443 blocked between nodes. On the agent itself, journalctl -u k3os and the k3s agent logs are where boot-time failures surface — though see the persistence caveat below.

Running from RAM: the trade-off that defined the system

k3OS could run largely from RAM with the root filesystem as an overlay. This made rollback trivially easy — reboot and you are back to the known-good image — and reduced wear on SD cards and cheap flash in edge hardware.

The cost is easy to overlook: anything not explicitly persisted vanishes on reboot. That includes logs, which makes debugging a boot failure genuinely painful, and any local state your workloads wrote outside a persistent volume. If you reproduce this setup, decide up front where logs go (a remote syslog target is the clean answer) and confirm your persistent data directories are on mounted, durable storage before you trust the cluster with anything.

The second cost was coupling. k3OS pinned a k3s version per OS release, so upgrading Kubernetes meant upgrading the OS image. That is actually a coherent design — one artifact, one tested combination — but it ties your cluster upgrade cadence to the distro's release cadence, and when the distro stopped shipping releases, the Kubernetes version froze with it.

What to use instead, and what to keep

Do not deploy k3OS in 2026; it is unmaintained and will not receive security fixes. The ideas, however, live on. SUSE's Elemental tooling applies the same declarative, image-based model to Rancher-managed edge clusters, and running k3s on a generic immutable or image-based distro (with configuration delivered via cloud-init or Ignition) gets you most of the same properties with a supported base.

The transferable lessons:

1. Declare cluster membership, don't perform it. If joining a cluster requires a human running a command, your fleet will drift. Put the token and server URL in provisioning data.
2. Treat day-2 changes as rebuilds. Editing config and rebooting (or reimaging) beats mutating a live system, because the file in version control is always the truth.
3. Decide what survives a reboot before you need it to. Ephemeral root filesystems are wonderful until your only copy of the failure logs disappears with the machine that failed.

If you want to evaluate the original firsthand, the archived repository and ISOs are still downloadable — boot one in a VM, write a minimal config.yaml, and watch a cluster assemble itself. Just keep it off anything that matters.