k3os Config Fetched at Boot: Local config.yaml Works, Production Node Boots With Defaults

Answer First

The most likely cause is that the production node is being told at boot time to use a remote config source via kernel command-line parameters (such as k3os.mode and its datasource URL), and that fetch is either failing or overriding the local file. k3os is designed to continue booting when a remote config is unreachable, so the node comes up with defaults instead of your hostname, SSH keys, and k3s registration arguments. This is a likely explanation, not a confirmed one — the checks below will confirm it.

Why Local Validation Didn't Catch This

A local VM test is not equivalent to the production path for two reasons:

• Different config source: A locally installed node typically reads /var/lib/rancher/k3os/config.yaml from disk. A PXE- or cloud-booted node may be directed to a remote URL by kernel parameters, in which case the on-disk file is not the source of truth for that boot.
• First-boot semantics: Some k3os settings apply only on first boot of a fresh disk or after explicit config reapplication. Editing config.yaml on an already-initialized node does not retroactively apply hostname, SSH, or k3s arguments.

Verification Steps

1. Determine which config source was selected. On the affected node, inspect the kernel command line:

   cat /proc/cmdline

   Look for k3os.mode and any datasource URL parameters. Their presence means the boot was not relying solely on the local file.
2. Compare on-disk config to intended config.

   diff /var/lib/rancher/k3os/config.yaml /path/to/intended-config.yaml

   Also check for any local.d overrides that may take precedence.
3. Check boot logs for fetch failures.

   journalctl -b | grep -i k3os

   Datasource fetch errors here distinguish "fetched the wrong source" from "fetch failed and continued with defaults." A serial console capture works if the journal is unavailable.
4. Test reachability of the remote URL from the node's boot network. Lab and production VLANs often differ in DHCP, DNS, or routing, so a URL reachable from your workstation may not be reachable during early boot on the production segment.

Pinning vs. Migrating

k3os is archived and unmaintained, with final releases in the 0.x line. Pinning the last release freezes known bugs and unpatched components; migrating to a maintained successor (for example, standard k3s on a conventional or immutable distribution) trades migration effort for ongoing security updates. For an isolated lab node, pinning is a defensible default. For anything internet-facing or multi-tenant, migrating is the safer operational choice. Exact datasource parameter names and fallback behavior vary across 0.x releases, so verify against the docs for your pinned version.

One Missing Detail

The full output of cat /proc/cmdline from the production node would confirm which datasource was selected and change the recommendation from "check both paths" to a specific fix.