Configuring K3s High Availability with Embedded etcd

Solving Control Plane Single-Points-of-Failure

By default, K3s uses SQLite for its data store, which limits the control plane to a single server node. If that node fails, the entire cluster's management layer goes offline. To achieve high availability (HA), you must replace SQLite with an embedded etcd cluster. This allows multiple server nodes to share the cluster state, ensuring that the API remains available even if a specific server node crashes.

The critical takeaway for this configuration is the quorum requirement: you must deploy an odd number of server nodes (typically three) to prevent "split-brain" scenarios, where the cluster cannot decide which node holds the truth.

Prerequisites

• Three or more Linux nodes with network connectivity.
• A load balancer (e.g., HAProxy, Nginx, or a cloud LB) configured to distribute traffic across the server nodes on port 6443.
• Root or sudo access on all participating nodes.
• A unique, secure cluster token string.

Step 1: Initialize the First Server Node

The first node must initialize the etcd database. Run the following command on your first server node. Replace <CLUSTER_TOKEN> with your chosen secret string.

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --token <CLUSTER_TOKEN>

Execution Context: Run as root on the first server node. The --cluster-init flag tells K3s to start an embedded etcd cluster rather than using SQLite.

Step 2: Join Additional Server Nodes

Once the first node is active, join the remaining server nodes to the cluster. You must point them to the load balancer endpoint or the first node's IP. Replace <FIRST_NODE_IP> and <CLUSTER_TOKEN> accordingly.

curl -sfL https://get.k3s.io | sh -s - server \
  --server https://<FIRST_NODE_IP>:6443 \
  --token <CLUSTER_TOKEN>

Execution Context: Run on each subsequent server node. The --server flag tells the node where to find the existing etcd cluster to synchronize state.

Infrastructure Comparison: SQLite vs. Embedded etcd

Feature	SQLite (Default)	Embedded etcd (HA)
Control Plane Nodes	Single Node	Multiple (Odd number recommended)
Failure Tolerance	None (Single point of failure)	Can lose (N-1)/2 nodes
Disk I/O Sensitivity	Low	High (Requires low-latency SSDs)
Setup Complexity	Minimal	Requires Load Balancer

Verification and Health Checks

After all nodes have joined, verify the cluster state from any server node using kubectl.

1. Check Node Status: Run kubectl get nodes. All server nodes should appear with a status of Ready.
2. Verify etcd Health: Check the system pods to ensure the etcd components are running: kubectl get pods -n kube-system.
3. Test Failover: Power off one of the server nodes. Attempt to run kubectl get pods -A via the load balancer. The API should remain responsive as long as a quorum (majority) of nodes is still online.

Operational Limitations and Risks

• Disk Latency: etcd is extremely sensitive to disk write latency. If you use slow HDDs or network-attached storage with high latency, you may experience frequent leader elections or cluster instability. Use SSDs for the /var/lib/rancher/k3s/server/db directory.
• Token Management: If the cluster token is changed after initialization, you must manually update the configuration on all existing nodes and restart the K3s service.

Rollback Procedure

If the HA setup fails or you wish to return to a single-node SQLite cluster, you must wipe the state from all nodes to avoid configuration conflicts.

1. Run the K3s uninstall script on all nodes: /usr/local/bin/k3s-uninstall.sh.
2. Manually remove any remaining data directories: rm -rf /var/lib/rancher/k3s/server/db.
3. Re-install the first node without the --cluster-init flag.