k3OS Automatic Cluster Join: Architecture, Trust Boundaries, and Operational Checks

Problem Statement

When deploying k3OS on edge devices, operators often want a node to join a k3s cluster automatically after boot without manual CLI steps. The built‑in join‑token mechanism can do this, but understanding its requirements, minimal design, trust boundaries, and failure modes is essential to avoid silent failures or security gaps.

Requirements for Automatic Join

• k3OS version 1.28+ (token format and TLS handling differ in older releases).
• Persistent storage for /var/lib/rancher/k3s/server/tls and /etc/rancher/k3s/k3s.yaml (stateless devices lose the token and certs on reboot).
• Network connectivity to the k3s API server on port 6443 (firewalls or NAT must allow TCP).
• Valid, non‑expired join token (single‑use, short‑lived string).
• CA certificate that matches the cluster’s root CA (fingerprint verification).

Minimal Design – The Systemd Unit

The core of the automatic join is the k3s.service unit. It pulls the k3s binary, starts containerd, and runs the agent with the token and certificates present in the file system.

[Unit]
Description=K3s Agent
After=network-online.target

[Service]
ExecStart=/usr/local/bin/k3s agent \
    --kubelet-arg=authentication-token-expiration=1h \
    --token-file=/etc/rancher/k3s/k3s.yaml \
    --tls-cert-file=/var/lib/rancher/k3s/server/tls/k8s.localhost.crt \
    --tls-key-file=/var/lib/rancher/k3s/server/tls/k8s.localhost.key
Restart=on-failure

[Install]
WantedBy=multi-user.target

Running systemctl enable --now k3s on the node starts the agent automatically.

Trust & Data Boundaries

• Token Boundary: The token is a short‑lived, single‑use string. It should never be stored in version control or exposed via logs. Only the node’s /etc/rancher/k3s/k3s.yaml holds it.
• CA Boundary: The CA cert in /var/lib/rancher/k3s/server/tls/ca.crt is the root of trust. The node validates the server’s certificate chain against this file during TLS handshake.
• Certificate Boundary: Node certs (k8s.localhost.crt and k8s.localhost.key) are used for mutual TLS. They are generated during initial installation and signed by the cluster CA.
• Network Boundary: The agent only contacts the API server over TLS on port 6443. No other external endpoints are reached automatically.

Operational Checks

1. Service Status
   Run on the node:

   sudo systemctl status k3s

   Expect: Active: active (running). Any failed state indicates a startup error.
2. API Reachability
   

   curl -k https://:6443/healthz

   Expect: ok. A non‑200 response suggests network or TLS misconfiguration.
3. Cluster Membership
   From a control plane or the node itself (if kubectl is installed):

   kubectl get nodes -o wide

   The node should appear with the role worker (or server if configured as a control plane).
4. Certificate Fingerprint
   Verify the node’s cert fingerprint against the cluster CA:

   openssl x509 -fingerprint -noout -in /var/lib/rancher/k3s/server/tls/k8s.localhost.crt
   openssl x509 -fingerprint -noout -in /var/lib/rancher/k3s/server/tls/ca.crt

   The fingerprints should match the CA’s fingerprint distributed to the node.
5. Token Presence
   Check /etc/rancher/k3s/k3s.yaml:

   cat /etc/rancher/k3s/k3s.yaml
   # Expected: token: <TOKEN>

   An empty or missing token field means the node cannot re‑join after a reboot.

Failure Modes

• Token Expiration: The token is single‑use and short‑lived. If the node reboots before the token is consumed, the join fails. Rotating the token and updating k3s.yaml is required.
• Network Partition: Firewalls or NAT that block TCP/6443 prevent the agent from contacting the API server. The node will remain in NotReady state.
• Corrupted TLS Files: Deleting or corrupting ca.crt, k8s.localhost.crt, or k8s.localhost.key stops authentication. The agent logs a TLS handshake error.
• CA Mismatch: If the node’s CA does not match the cluster’s root CA, the TLS handshake fails. This can happen when moving a node between clusters without updating the CA.
• Mis‑configured Service Unit: Wrong ExecStart arguments or missing --token-file cause the agent to start without a token, leading to a failed join.

When Design Changes Are Needed

While the minimal design works for a single node or a small edge deployment, several scenarios warrant a redesign:

• Scaling to Many Edge Nodes: Using a shared token across many nodes reduces isolation. Per‑node tokens (generated via k3s token create on the control plane) provide better security and easier revocation.
• External Etcd Store: If the cluster uses an external etcd, the node must be configured with the etcd endpoints and TLS certificates. The unit would need additional --cluster-servers arguments.
• Custom CNI with Pre‑Configured Policies: Some CNIs (e.g., Calico with IPIP) require node labels or annotations before the agent starts. A pre‑boot script that writes these labels into /etc/rancher/k3s/k3s.yaml or uses --node-labels may be necessary.
• Stateless Devices: Devices without persistent storage lose the token and certs on reboot. In this case, a bootstrap script that pulls fresh credentials from a secure vault (e.g., HashiCorp Vault) during boot is required.
• High Availability Control Plane: When multiple control planes exist, the agent must be able to fail over to another API endpoint. The unit would need --server https://api1:6443,https://api2:6443 and health checks for each.

Practical Verification Checklist

Check	Command	Expected Result
Service Running	sudo systemctl status k3s	Active: active (running)
API Health	curl -k https://:6443/healthz	ok
Node in Cluster	kubectl get nodes	Node listed with role worker/server
Token Present	cat /etc/rancher/k3s/k3s.yaml | grep token	token: <TOKEN>
CA Fingerprint	openssl x509 -fingerprint -noout -in /var/lib/rancher/k3s/server/tls/ca.crt	Matches cluster CA fingerprint

Conclusion

The k3OS automatic join feature is a lightweight, zero‑touch solution for edge deployments, but it hinges on correct token and certificate handling, network reachability, and persistent storage. By following the operational checks and being aware of failure modes, operators can confidently scale their clusters or adapt the design when additional requirements arise.