Deploying a High‑Availability k3OS Cluster with Built‑In etcd: A Practical Guide

Why Use k3OS HA with Embedded etcd?

Running Kubernetes in production often means you need a resilient control plane that can survive node failures. For small to medium deployments, k3OS offers an out‑of‑the‑box solution: when you launch a cluster in HA mode, it automatically spins up a three‑node embedded etcd cluster and deploys the API server as a static pod. This eliminates the need to hand‑craft etcd manifests, TLS certificates, or external etcd provisioning. The trade‑off is a fixed 3‑node limit and the static pod mechanism, but for many edge or edge‑to‑cloud scenarios the convenience outweighs the constraints.

Getting Started: Creating the HA Cluster

Below is a step‑by‑step walkthrough that assumes you have three bare‑metal or virtual machines ready. All commands run on the first node; the others will be joined automatically.

1. Install k3OS on each node (use the ISO or ISO‑less installer). After boot, SSH into the first machine.

2. Create the HA cluster with the built‑in etcd:

   sudo k3os cluster create --ha
   

   Explanation:

   • --ha tells k3OS to generate a three‑node etcd cluster.
   • The command writes the cluster configuration to /etc/k3os/cluster.yaml and deploys the etcd static pod.
   • k3OS automatically generates TLS certificates for the API server and etcd communication.

3. On the other two nodes, run k3os join to add them to the cluster. The join token is printed after the first node’s command finishes.

   sudo k3os join https://<first-node-ip>:6443 --token <token>
   

4. Verify that the cluster is healthy. From any node, run:

   kubectl get nodes
   

   You should see three nodes in Ready state.

5. Check that the embedded etcd service is active:

   systemctl status k3os-etcd
   

Inspecting the Auto‑Generated Configuration

k3OS writes the cluster definition to /etc/k3os/cluster.yaml. Open it to see the etcd endpoints, TLS settings, and static pod manifests. A snippet looks like this:

apiVersion: v1
kind: Config
clusters:
- name: cluster
  cluster:
    server: https://<first-node-ip>:6443
    certificate-authority-data: <base64‑encoded‑ca>
nodes:
- address: <first-node-ip>
  role: controlplane
- address: <second-node-ip>
  role: controlplane
- address: <third-node-ip>
  role: controlplane
etcd:
  servers:
  - https://<first-node-ip>:2379
  - https://<second-node-ip>:2379
  - https://<third-node-ip>:2379
  caFile: /etc/k3os/ca.crt
  certFile: /etc/k3os/etcd.crt
  keyFile: /etc/k3os/etcd.key

Notice that the etcd section references the same TLS files used by the API server, ensuring secure communication.

Rolling Upgrades: Keeping the API Server Alive

k3OS performs upgrades by pulling a new image and restarting the static pod. Because the API server is part of a rolling update strategy, at least two nodes stay online during the process. Test it with:

sudo k3os upgrade

While the upgrade runs, run kubectl get nodes on a separate terminal. You should see nodes coming online one after another, and the cluster never loses the Ready status on all three nodes.

Trade‑Offs and Limitations

• 3‑node limit: The built‑in etcd cluster supports only three control plane nodes. For larger clusters, you must either manually configure etcd or point k3OS to an external etcd cluster via the --etcd-endpoints flag.
• Static pod manifests: k3OS stores the etcd and API server manifests in /etc/k3os/static-pods. If you need to tweak etcd parameters (e.g., data directory, snapshot frequency), you’ll have to edit the manifests and restart the service manually.
• Certificate rotation: k3OS auto‑generates certificates on first boot. If you need to replace them (e.g., after a CA change), you must regenerate the cluster configuration and re‑apply the manifests.

When to Use External etcd

If you anticipate scaling beyond three control planes or require a dedicated etcd cluster for compliance, you can still use k3OS with an external etcd endpoint. Create a cluster with --etcd-endpoints pointing to your etcd cluster and provide the CA, cert, and key files. The rest of the bootstrap process remains unchanged.

Actionable Checklist

1. Provision three nodes with k3OS installed.
2. Run k3os cluster create --ha on the first node.
3. Join the remaining nodes with k3os join.
4. Verify cluster health with kubectl get nodes and systemctl status k3os-etcd.
5. Optional: Test a rolling upgrade with k3os upgrade.
6. Document the /etc/k3os/cluster.yaml file for future reference.
7. If scaling, plan an external etcd deployment and re‑create the cluster with the appropriate flags.

By following this guide, you can quickly spin up a resilient k3OS cluster with minimal configuration overhead, while being aware of the 3‑node constraint and static pod management.