k3os config.yaml applies locally but k3s joins with stale token on production cold boot

Direct answer

You are seeing a boot-ordering race, not a config parsing failure. On a cold disk install, k3s (or k3s-agent) can start from its systemd enablement symlink before the k3os config stage has finished writing the token derived from config.yaml. The agent then reads whatever token file already exists on disk — from a previous boot, an earlier install attempt, or an empty/default state — and joins with that stale value. Your local VM never reproduces it because iterative reboots happen on an already-installed system where the token file from the prior cycle is correct, so even a "wrong" ordering produces the right result.

Likely explanation vs. confirmed facts

Confirmed behavior: k3os processes /var/lib/rancher/k3os/config.yaml (or a boot datasource such as a remote URL or kernel cmdline) during early boot stages, and the k3s unit is a normal systemd service ordered after local-fs.target / network-online.target. Nothing in that ordering strictly guarantees the config-write step has completed first.

Likely, not confirmed: on production cold boots the datasource differs (cloud metadata or remote URL fetched over a slow/isolated VLAN), so the config stage completes later than on your local VM, widening the race window. DHCP delays on an isolated VLAN make this worse because a URL-based datasource cannot even be fetched until the link is up.

How to confirm on a production node

Compare the token file's timestamp against the service start time:

stat /var/lib/rancher/k3s/server/token 2>/dev/null || stat /var/lib/rancher/k3s/token
journalctl -u k3s-agent.service -u k3s.service --no-pager | head -40
systemd-analyze plot > boot.svg   # if you can pull the file off

If the unit's "Started" line precedes the token file's mtime, the race is confirmed. For headless nodes, capture early boot via serial console or netconsole and grep for the config-write messages versus the k3s start line. Also diff the effective config: on each node run k3os config (or dump /var/lib/rancher/k3os/config.yaml plus any datasource URL) via your provisioning pipeline and compare hashes between the VM and the production node — a datasource that silently fails to fetch leaves you with a partial or default config.

Scoped mitigations

1. Order the unit after config application. Add a drop-in such as /etc/systemd/system/k3s-agent.service.d/after-config.conf with After=k3os-config.service (adjust to the actual unit name on your release — verify with systemctl list-units | grep k3os) and ConditionPathExists=/var/lib/rancher/k3s/token so the agent refuses to start on an empty token rather than joining wrongly.
2. Pin the datasource locally. Prefer a config baked into the install (kernel cmdline or on-disk file) over a remote URL on networks with slow DHCP.
3. Match server/agent versions. k3os pins k3s to its release; a token generated against a different server version fails in ways that look like a stale token. Verify with k3s --version on both sides.

Migration note

k3os is archived, so any ordering workaround is fragile against future changes and receives no fixes. The lowest-friction path that preserves your existing config contract is Elemental (or SLE Micro) with cloud-init: your SSH keys, network files, and k3s token move into a standard cloud-config, and k3s runs as a regular systemd unit you fully control — including explicit After= ordering. Generic Linux + k3s with cloud-init is equally viable if you want to drop the OS-management layer entirely. Treat this section as a planning direction and verify current project status before committing.

One detail that would change this advice

Which datasource does production use — on-disk file, kernel cmdline, or remote URL? If it is a remote URL, the fix is datasource reliability (or pre-baking the config), not systemd ordering.