Building a Minimal High‑Availability k3OS Cluster with Built‑In etcd

Problem Statement

You need a lightweight, highly available Kubernetes distribution that can run on bare metal or virtual machines without a separate etcd operator. k3OS offers a bundled k3s + etcd HA mode, but the documentation is terse. This article walks through the exact steps, trust boundaries, and operational checks required to deploy a 3‑node k3OS cluster that remains available if any single node fails.

Requirements

• Three identical hosts (VMs or bare‑metal) with at least 2 GB RAM and 20 GB root disk.
• Static IP addresses: <NODE1_IP>, <NODE2_IP>, <NODE3_IP>.
• Open TCP ports 6443 (k8s API), 2379–2380 (etcd), 80/443 (optional load balancer).
• Root or sudo access on each host.

Minimal Design

k3OS can self‑manage an etcd cluster when the first node is started with --cluster-init and subsequent nodes with --cluster-reset pointing back to the first node. The cluster runs k3s on each host; the etcd data lives on the root filesystem. A simple TCP load balancer (e.g., HAProxy or MetalLB) is recommended to expose a single stable API endpoint.

Step‑by‑Step

1. Install k3OS on all nodes. Download the latest k3os.iso and boot each host.
2. Configure the first node (Node 1).

   # On Node 1
   k3os install --cluster-init
   # The installer will generate a kubeconfig at /etc/rancher/k3s/k3s.yaml
   

3. Configure the remaining nodes (Node 2 & 3).

   # On Node 2
   k3os install --cluster-reset --cluster-endpoints=<NODE1_IP>
   # On Node 3
   k3os install --cluster-reset --cluster-endpoints=<NODE1_IP>
   

4. Verify cluster state.

   # From any node after all three are up
   kubectl get nodes
   # Expected output: three Ready nodes
   

5. Set up a load balancer. Example HAProxy config:

   frontend k8s
       bind *:6443
       mode tcp
       default_backend k8s-backend
   
   backend k8s-backend
       mode tcp
       balance roundrobin
       server node1 <NODE1_IP>:6443 check
       server node2 <NODE2_IP>:6443 check
       server node3 <NODE3_IP>:6443 check
   

Configuration Summary

Parameter	Value
etcd endpoints	https://<NODE1_IP>:2379,https://<NODE2_IP>:2379,https://<NODE3_IP>:2379
etcd TLS	enabled via k3OS built‑in cert manager
API endpoint	Load balancer IP:6443

Trust & Data Boundaries

• etcd cluster is the sole source of truth. All control‑plane data, including secrets, lives here.
• TLS certificates are generated by k3OS’s cert manager and stored under /var/lib/rancher/k3s/server/tls/. They must remain confidential; rotate them every 90 days with k3s cert renew.
• Root filesystem is the storage backend for etcd. Monitor its size to avoid a read‑only state.

Operational Checks

• etcd health:

  etcdctl --endpoints=https://<NODE1_IP>:2379,https://<NODE2_IP>:2379,https://<NODE3_IP>:2379 endpoint health
  

• Node status:

  kubectl get nodes -o wide
  

• Disk usage:

  df -h /var/lib/rancher/k3s/server/db
  

• Automate these checks with Prometheus exporters or simple cron jobs.

Failure Modes & Mitigations

• Node failure: If any node stops, the remaining two keep quorum. Verify by stopping k3s on Node 2 and observing that workloads continue.
• etcd disk full: The cluster becomes read‑only. Mitigation: set up a separate persistent volume for etcd or enforce a disk quota on the root filesystem.
• Load balancer misconfiguration: API traffic may drop. Use health checks and keep the LB monitor separate from the cluster.
• Certificate expiration: Services may fail to authenticate. Automate renewal and re‑apply certificates on all nodes.

When to Re‑Design

• More than 5 nodes: k3OS’s built‑in etcd cluster is not designed for large scale. Consider an external etcd operator.
• High write loads: etcd on the root filesystem may become a bottleneck. Migrate to a dedicated etcd storage backend.
• Regulatory compliance: if you require strict isolation, move etcd to a separate, hardened host.

Concrete Example: Simulating a Node Failure

To confirm HA, stop k3s on Node 2:

# On Node 2
sudo systemctl stop k3s

Run kubectl get nodes from the load balancer IP. You should still see Node 1 and Node 3 as Ready, and workloads should continue to schedule. Restart Node 2 with sudo systemctl start k3s and verify it rejoins the cluster.

Check etcd health again to ensure quorum is intact.

Practical Verification Checklist

• All nodes report Ready in kubectl get nodes.
• etcd health shows true for each peer.
• TLS certificates are present and not expired.
• Root disk usage < 80 % to avoid read‑only state.
• Load balancer reports all backends as healthy.

Maintain these checks in a CI/CD pipeline or monitoring stack to ensure the cluster remains highly available.