k3OS provisioning: cloud-init versus installer ISO

Decision: Provision k3OS nodes with cloud-init or the installer ISO

You need to deploy k3OS nodes that join an existing k3s cluster, but k3OS keeps its root filesystem read-only and managed by OSTree. You cannot log in after installation and edit system files expecting changes to survive an atomic OS update. The practical decision is therefore how to inject configuration before the node first boots: automate it with cloud-init, or install interactively with the k3OS ISO.

Constraints that shape the decision

• Immutable root: k3OS uses OSTree to manage a read-only root partition. Atomic OS upgrades replace the entire root deployment, so manual edits outside OSTree are lost.
• Signed boot: The OS image is verified at startup; unsigned modifications are rejected.
• No custom kernel modules: The immutable root does not support loading out-of-tree kernel modules.
• Default networking: Canal (Calico plus Flannel) provides the cluster network; competing overlay plugins may conflict.

Supported options compared

Factor	cloud-init	Installer ISO
Best suited for	Large fleets, virtual machines, remote hosts	Bare-metal servers, one-off nodes, air-gapped hardware
Configuration timing	Applied at first boot from user-data	Entered during interactive installation
k3s join details	Token and server URL in user-data	Token and URL prompted during install
Network configuration	Defined in cloud-init network config	DHCP by default; static IP via installer or kernel parameters
Scalability	High; templates can be reused	Low; each node requires manual media
Root filesystem impact	Preserves OSTree immutability	Preserves OSTree immutability

Trade-offs

cloud-init scales well because the same user-data can provision many nodes, but it requires a datasource the node can reach on first boot. A syntax error or wrong token in user-data prevents the node from joining the cluster, and because the root is immutable, you cannot simply SSH in and fix the file afterward.

The installer ISO is straightforward for a single server and works without network automation infrastructure, but it does not scale. It is most useful when you need to validate hardware or install a node that cannot reach a cloud-init datasource.

Implementation: Automating first boot with cloud-init

This example shows a representative cloud-init user-data file for a k3OS node. The exact field names depend on your k3OS release; validate the schema against your version before deploying.

Where to run: On your provisioning host or cloud platform before the node's first boot.

Permissions: Root or sudo on the provisioning host to create the user-data resource. No k3OS credentials are needed before boot.

Placeholders: Replace <K3S_URL>, <K3S_TOKEN>, <STATIC_IP>, <GATEWAY>, and <DNS> with values from your cluster.

#cloud-config
hostname: k3os-node-01

ssh_authorized_keys:
  - ssh-rsa AAAA... [contact removed]

write_files:
  - path: /etc/rancher/k3os/config.yaml
    permissions: '0644'
    content: |
      token: <K3S_TOKEN>
      server: <K3S_URL>
      network:
        eth0:
          dhcp: false
          address: <STATIC_IP>/24
          gateway: <GATEWAY>
          nameservers:
            - <DNS>

Attach this user-data to the virtual machine or instance before powering it on. k3OS reads the configuration during first boot, configures the network, and attempts to join the k3s server specified in server.

Risks

• An incorrect token or server value causes the node to fail joining the cluster. Reprovisioning with corrected user-data is usually faster than trying to repair an immutable root.
• Do not deploy a competing CNI plugin alongside Canal unless you have tested for conflicts; the default cluster-wide policy assumes Canal is the only network plugin.
• If your workload requires a custom kernel module, k3OS cannot load it on the immutable root. Choose a different OS or redesign the workload to run entirely in userspace containers.

Validation: Confirm the node joined and the OS is intact

After the node boots, perform these checks from the node itself or from your management workstation.

On the k3OS node

Command: rpm-ostree status

Where to run: Local shell on the node (console or SSH).

Permissions: Root or sudo.

Expected check: Output lists the current OSTree deployment and its checksum. This confirms the node is booted from a signed, OSTree-managed image.

From the management workstation

Command: kubectl get nodes -o wide

Where to run: Workstation with kubeconfig access to the cluster.

Permissions: Cluster read access.

Expected check: The new node appears in the list with status Ready and internal IP matching your cloud-init network config.

Boot integrity

Review early boot output (serial console or virtual machine console) for confirmation that the OS image signature was verified. If the signature check fails, the node will not complete boot; do not bypass this check.

Limitations

• OSTree atomic updates replace the root filesystem. Persistent application data must live outside the root, such as in Kubernetes persistent volumes or bind-mounted data partitions.
• Custom kernel modules are unsupported on the immutable root.
• Overlay network plugins may conflict with the default Canal configuration.

Practical result to check

A successful deployment ends with the node listed as Ready in kubectl get nodes and rpm-ostree status reporting a valid OSTree deployment. If either check fails, reprovision with corrected user-data rather than editing the running system.

Note: This guide describes well-established k3OS behavior. Verify exact cloud-init field names and OSTree command output against your installed k3OS version before using in production.