Zero‑Touch Provisioning of k3OS Nodes with a User‑Supplied cloud‑config

Desired outcome

Automatically boot a k3OS node, apply a user‑provided cloud‑config file, and have the node join a existing k3s cluster as an agent without manual login or post‑boot configuration.

Prerequisites

• A bootable k3OS image (ISO, raw disk, or cloud image) – the guide assumes version v0.20.0, which bundles a specific k3s release.
• A reachable k3s server endpoint (e.g., https://server.example.com:6443) with a valid token for agent nodes.
• A cloud‑config file in YAML format accessible via HTTP, local disk, or embedded in the image.
• Basic networking (DHCP or static IP) so the node can reach the k3s server.

Procedure

1. Create the cloud‑config

   Define the hostname, SSH keys, network settings, and the k3s token. The file must be valid YAML and placed where k3OS can read it during early init.

   # Example cloud‑config (save as conf.yaml)
   hostname: k3os-node-01
   ssh_authorized_keys:
     - ssh-rsa AAAAB3NzaC1yc2E... [contact removed]
   network:
     interfaces:
       eth0:
         dhcp: true
   k3s:
     token: K10s...example-token
     # K3S_URL is read from the environment; you can also set it here if desired
     # extra_env: []
   
   Place the file at /var/lib/k3os/conf.yaml on the boot medium (e.g., copy it into the ISO’s overlay directory) or make it available via an HTTP server that the node can reach.
   

2. Boot the k3OS image

   Start the node from the prepared image. During early init, k3OS mounts a read‑only squashfs root, overlays a writable /var/lib/k3os directory, and applies the cloud‑config found at /var/lib/k3os/conf.yaml.

3. Wait for the k3s agent to start

   The cloud‑config triggers k3OS to launch k3s in agent mode, using the K3S_URL environment variable (or the token from the config) to join the cluster.

4. Verify from the node

   After boot, you can check the agent service:

   systemctl status k3s-agent

   Look for active (running) in the output.

5. Verify from the k3s server

   On the server, run:

   kubectl get nodes

   The new node should appear with status Ready. You can also compare versions:

   k3s agent --version

   Ensure it matches the server’s k3s version (or is within the compatible range documented for the k3OS release).

Expected checks

• systemctl is-active k3s-agent returns active.
• The node’s hostname matches the value set in cloud‑config.
• cat /var/lib/k3os/conf.yaml shows the exact YAML you supplied, confirming the overlay was applied.
• From the server, kubectl get nodes -o wide lists the node and its internal IP.

Recovery options

If the cloud‑config is malformed or the node cannot reach the k3s server:

• k3OS drops to a rescue shell on the console. You can examine logs with journalctl -u k3s-agent to see the error.
• Correct the cloud‑config (fix YAML syntax, verify token, ensure network connectivity) and either:
• Replace /var/lib/k3os/conf.yaml in the overlay and reboot, or
• Re‑boot the node with a corrected image (re‑create the ISO/raw image with the fixed file).
• Persistent state (e.g., logs, container images) remains in /var/lib/k3os across reboots, so a corrected config will not lose that data.
• If the k3s version bundled in k3OS is incompatible with your server, you must either upgrade the k3s server to a compatible version or rebuild k3OS with a newer k3s bundle (outside the scope of zero‑touch provisioning).

Limitations and practical verification

k3OS v0.20.0 is no longer actively maintained; security updates for newer kernels or k3s releases may be missing. Before using in production, verify that the bundled k3s version meets your cluster’s version‑skew policy.

To confirm that the node’s root filesystem is indeed read‑only except for the overlay, run:

mount | grep ' / '

You should see a line indicating ro for the squashfs device and a separate line for /var/lib/k3os mounted as rw.

Finally, a simple end‑to‑end test is to deploy a test pod on the new node:

kubectl run nginx --image=nginx --restart=Never --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"k3os-node-01"}}}'

Check that the pod starts and can be accessed via a NodePort or ClusterIP service.