Building a Three-Node K3s HA Cluster with Embedded etcd

The Problem: A Single K3s Server Is a Single Point of Failure

A default K3s server stores cluster state in SQLite on one node. If that node fails, the Kubernetes API server stops responding. Existing pods keep running, but you cannot create workloads, rotate secrets, or recover failed nodes until the server returns.

The useful takeaway is that K3s can replace SQLite with embedded etcd. Running an odd number of server nodes—typically three—creates a quorum. The API stays available as long as a majority of control-plane nodes are healthy.

Desired Outcome

Three K3s server nodes running embedded etcd, reachable through one load-balanced endpoint on port 6443. The cluster should remain operable after any one server node is stopped.

Prerequisites

• Three Linux servers with root or sudo access.
• SSD or NVMe storage for /var/lib/rancher/k3s/server/db; etcd is sensitive to disk latency.
• Network access between servers on TCP 6443, 2379, and 2380.
• A load balancer or DNS record pointing to all three server IPs on port 6443.
• K3s version installed identically on every node.

Procedure

1. Initialize the first server with embedded etcd

Run as root on the first server node. Replace <LB_IP_OR_DNS> with your load balancer address.

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --tls-san <LB_IP_OR_DNS>

--cluster-init starts embedded etcd and marks this node as the cluster seed. --tls-san adds the load balancer address to the API server certificate so clients do not see TLS name mismatches.

2. Retrieve the node token

Run on the first server node:

cat /var/lib/rancher/k3s/server/node-token

Copy the printed token. Every joining server must present the same shared secret.

3. Join the remaining server nodes

Run as root on each additional server. Replace <FIRST_NODE_IP>, <NODE_TOKEN>, and <LB_IP_OR_DNS>:

curl -sfL https://get.k3s.io | sh -s - server \
  --server https://<FIRST_NODE_IP>:6443 \
  --token <NODE_TOKEN> \
  --tls-san <LB_IP_OR_DNS>

Repeat for the third node. Do not add more than five server nodes; each additional etcd member increases consensus overhead.

4. Point the load balancer at the servers

Configure TCP port 6443 to all three K3s server IPs. Use an HTTP or TCP health check against /healthz on port 6443 if your load balancer supports it.

Expected Checks

Verify all servers are Ready

Run on any node or from a machine with kubectl configured:

kubectl get nodes

Expected result: all three servers show Ready with control-plane roles.

Verify etcd pods are running

kubectl get pods -n kube-system -l component=etcd

Expected result: one etcd pod per server node, all in Running state.

Verify the API through the load balancer

kubectl --server=https://<LB_IP_OR_DNS>:6443 get nodes

Expected result: the node list returns without certificate errors.

Test single-node failure

Stop K3s on one server (systemctl stop k3s), then repeat the API check through the load balancer. The API should remain responsive because two of three nodes still hold quorum.

Limitations

Factor	Limitation	Mitigation
Write performance	More than five etcd members slows consensus writes.	Use three or five servers; scale workers separately.
Disk latency	Slow storage triggers etcd leader-election timeouts.	Use SSD/NVMe for the etcd data directory.
Quorum loss	Losing two of three nodes makes the cluster read-only.	Spread servers across availability zones or hosts.

Rollback to Single-Node SQLite

Embedded etcd and SQLite are incompatible. Reverting requires wiping the server state and reinstalling.

1. Stop K3s on every server: systemctl stop k3s.
2. Run the uninstall script: /usr/local/bin/k3s-uninstall.sh.
3. Remove stale etcd data: rm -rf /var/lib/rancher/k3s/server/db.
4. Reinstall K3s without --cluster-init.

This procedure deletes cluster state. Back up any critical resources first.