Configuring k3OS with Cloud‑Config: Declarative OS and k3s Setup at Boot

Why Cloud‑Config Matters for k3OS

k3OS is a lightweight OS that ships with an embedded k3s Kubernetes distribution. Its root filesystem is read‑only, so any persistent change must be expressed declaratively. The cloud‑config file, processed by cloud‑init during early boot, is the single source of truth for both the host OS and the k3s cluster. By updating this file you can alter networking, users, SSH keys, and k3s arguments without rebuilding an image.

How k3OS Applies Cloud‑Config

• The system mounts a cloud‑config drive or reads the user‑data field from a cloud provider’s metadata service.
• During init, cloud‑init parses the YAML and executes supported modules: network, users, ssh_authorized_keys, write_files, runcmd, etc.
• For k3s, k3OS writes the final service configuration to /etc/rancher/k3s/config.yaml (or /etc/rancher/k3s/agent.yaml for agents) before launching the k3s service.
• If the cloud‑config changes, the next boot re‑applies the new settings, enabling immutable‑infrastructure workflows.
• When no cloud‑config is supplied, k3OS defaults to DHCP on all interfaces, creates a rancher user with no password, and starts k3s with its built‑in defaults.

A Worked Cloud‑Config Example

The following YAML demonstrates a typical k3OS configuration that:

• Sets a static IP on eth0.
• Adds a privileged user alice with an SSH key.
• Writes a custom k3s server configuration file.
• Runs a simple command after the system is fully booted.

#cloud-config

# 1. Network
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      addresses: [192.168.1.10/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]

# 2. Users
users:
  - name: alice
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy... [contact removed]

# 3. Write the k3s server config
write_files:
  - path: /etc/rancher/k3s/config.yaml
    content: |
      server: https://my-control-plane:6443
      token: abcdef123456

# 4. Run a command after boot (e.g., echo a message)
runcmd:
  - echo \"k3s will start with the custom config above\" | systemd-cat -p info

Place this file on a cloud‑config drive when creating a VM, or paste it into the user‑data field of a cloud provider’s metadata console. The file must be valid YAML; otherwise cloud‑init will log a warning and fall back to defaults.

Verifying the Result

1. After the node boots, check the k3s configuration:

   sudo cat /etc/rancher/k3s/config.yaml
   

2. Confirm that the network interface has the static address:

   ip addr show eth0
   

3. Verify that the alice user exists and can SSH (using the key provided):

   ssh alice@192.168.1.10
   

4. Check that k3s reports the node as ready:

   kubectl get nodes
   

5. Inspect cloud‑init logs for any errors:

   journalctl -u cloud-init | grep -i k3os
   

To test immutability, create a file outside the overlay, e.g., sudo touch /etc/test, reboot, and then verify that the file no longer exists. This confirms that only cloud‑config‑driven changes persist.

Limits and Common Mistakes

• Read‑Only Root FS: Direct edits to system files (e.g., apt‑get install, manual edits in /etc) are discarded on reboot. All desired state must be expressed in cloud‑config or an overlay layer.
• YAML Indentation: Cloud‑config is strict about indentation. A single misplaced space can cause the entire file to be ignored. Use a YAML linter before deployment.
• Unsupported Modules: k3OS supports a subset of cloud‑init modules. Modules like apt or yum are not available; attempting to use them will result in silent failures.
• Missing write_files Path: If the path for the k3s config file is wrong, k3s will start with default arguments. Verify the path matches /etc/rancher/k3s/config.yaml for servers.
• Overriding Defaults: When no cloud‑config is provided, the system falls back to DHCP and a minimal k3s install. If you rely on static networking or custom k3s flags, always supply a config.
• Reboot Required: Changes to cloud‑config take effect only on the next boot. If you need a hot‑reload, you must restart the cloud‑init and k3s services manually, but the immutable overlay still applies.
• Logging: Errors are recorded in /var/log/cloud-init.log. Inspecting this file early in troubleshooting saves time.

When to Use Cloud‑Config Over Manual Setup

Use cloud‑config when you need repeatable, version‑controlled node provisioning, especially in environments where images are immutable. It simplifies drift prevention and aligns with GitOps practices: store the YAML in a repository, apply it via a CI pipeline, and let k3OS enforce the desired state on every boot.