Diagnosing k3OS Boot Failures Due to k3os-config Issues

Recognizable condition

After powering on a k3OS node, the system boots to a login prompt but the k3s service never appears in systemctl output. Journal entries show messages such as "failed to load k3os-config" or "k3s: failed to start".

Cause / diagnostic table

Observed symptom	Likely cause
Boot logs contain "yaml: unmarshal errors"	Invalid YAML syntax in k3os-config
Logs show "config not found" or empty config section	Boot medium did not provide the config file or it was placed at the wrong path
k3s starts but immediately exits with "permission denied" on /var/lib/rancher/k3s	Root filesystem remounted as read‑only due to a failed overlay, indicating the immutable layer could not be verified

Ordered checks

1. Verify boot medium
   • On the machine used to create the boot USB/cloud‑init, confirm that a file named k3os-config.yaml exists in the root of the medium.
   • For USB, mount it and run ls /mnt/usb/k3os-config.yaml.
   • For cloud‑init, inspect the user‑data payload for the write_files entry that places the config at /k3os/config.yaml.
2. Validate YAML syntax
   • Run a YAML linter on the config file, e.g. yamllint k3os-config.yaml.
   • Check that required top‑level keys (hostname, k3s, ssh_key) are present and correctly indented.
3. Inspect early boot logs
   • After boot, access the console or serial output and run journalctl -b -k | grep -i k3os.
   • Look for lines like "Loading k3os-config from …" and note any error messages.
4. Test immutability
   • As root, try to create a temporary file: touch /usr/local/test.
   • If the command succeeds, the overlay is writable (unexpected); if it fails with "Read‑only file system", the immutable layer is active as intended.
5. Confirm k3s launch
   • Run systemctl status k3s.
   • If inactive, check journalctl -u k3s for start‑up errors.

Fixes tied to findings

Invalid YAML syntax

Correct the YAML file on the boot medium. A minimal valid example:

# k3os-config.yaml
hostname: node01
k3s:
  token: "shared-secret"
  extra_args:
    - --disable=traefik
ssh_key: |
  ssh-rsa AAAAB3NzaC1yc2E... [contact removed]

After editing, rebuild the boot image (or replace the USB file) and reboot.

Config not found / wrong path

Ensure the bootloader passes the config correctly:

• For BIOS/UEFI USB boot, add the kernel parameter k3os.config=url:///k3os-config.yaml to the boot entry.
• For cloud‑init, verify that the write_files section defines path: /k3os/config.yaml and content: | followed by the YAML.

Re‑create the boot medium with the corrected parameter or payload.

Immutable layer verification failure

If the root filesystem cannot be remounted read‑only, the underlying squashfs image may be corrupted.

• Re‑download the official k3OS ISO/image for your architecture.
• Verify its SHA256 checksum against the publisher’s release page.
• Re‑write the boot medium using a tool like dd or Rufus.

Escalation criteria

Proceed to further support when:

• The node repeatedly fails to load k3os-config after verifying the boot medium and YAML syntax.
• Immutable layer tests show a writable root filesystem despite using a verified image.
• k3s start‑up logs indicate missing kernel modules or hardware incompatibilities that are not resolved by standard configuration.

In these cases, collect the full boot journal (journalctl -b > boot.log) and the exact k3os-config used, then open a ticket with the k3OS maintainers or your distribution’s support channel.