Deploying a Three‑Node k3s Cluster with Embedded etcd HA

Why a Three‑Node k3s Cluster?

When you need a lightweight Kubernetes distribution that can survive a single node failure, k3s’ embedded etcd offers a turnkey solution. The key is to start the first server with --cluster-init and then add two more servers. An odd number of server nodes guarantees an etcd quorum, keeping the API server alive even if one node goes down.

Step‑by‑Step Setup

1. Prepare the first server (e.g., node1.example.com). Install k3s and bootstrap the cluster:

   sudo k3s server --cluster-init \
     --tls-san node1.example.com \
     --advertise-address 10.0.0.1
   

   Replace 10.0.0.1 with the node’s internal IP. The --tls-san flag allows the API server to be reachable via the host name.

2. After the first server is up, retrieve the join token:

   sudo cat /var/lib/rancher/k3s/server/node-token
   

   Copy the token; you’ll need it on the other servers.

3. On the second server (node2.example.com), start k3s as a server and join the cluster:

   sudo k3s server \
     --server https://10.0.0.1:6443 \
     --token <token> \
     --tls-san node2.example.com \
     --advertise-address 10.0.0.2
   

4. Repeat the same command on the third server (node3.example.com), changing the advertise address and SAN accordingly.

Under the Hood

With --cluster-init the first server creates a new etcd cluster. Subsequent servers join as etcd members. The embedded etcd listens on two ports:

• Client API: 2379 (used by etcdctl and the k3s API server)
• Peer communication: 2380 (used between etcd nodes)

The Kubernetes API server runs on 6443. All agents must reach at least one server on this port. Ensure that firewalls allow TCP traffic on 2379, 2380, and 6443 between the three servers.

Verifying the Cluster

On any server, run:

kubectl get nodes

You should see three nodes with status Ready. To confirm etcd health:

sudo etcdctl --endpoints https://127.0.0.1:2379 \
  --cacert /var/lib/rancher/k3s/server/tls/etcd/ca.crt \
  --cert /var/lib/rancher/k3s/server/tls/etcd/server.crt \
  --key /var/lib/rancher/k3s/server/tls/etcd/server.key \
  endpoint health --cluster

All three members should return healthy. Finally, shut down one server and verify the API remains reachable:

kubectl get nodes

The remaining two nodes stay Ready, proving the quorum works.

Backup and Restore

etcd snapshots are taken with etcdctl snapshot save:

sudo etcdctl --endpoints https://127.0.0.1:2379 \
  --cacert /var/lib/rancher/k3s/server/tls/etcd/ca.crt \
  --cert /var/lib/rancher/k3s/server/tls/etcd/server.crt \
  --key /var/lib/rancher/k3s/server/tls/etcd/server.key \
  snapshot save /backup/etcd-snap-$(date +%F).db

Restore by stopping the cluster, running k3s server --cluster-reset with the snapshot path, and restarting.

Common Pitfalls

• Even number of servers: a 2‑node cluster cannot keep the API running after one node fails because etcd requires a majority (2 out of 3).
• Missing peer ports: blocking 2380 between servers breaks etcd replication.
• Mixing SQLite and etcd: a node started with SQLite (default for single‑node installs) cannot join an etcd cluster. All servers must be initialized with --cluster-init or --server and share the same backend.
• Ignoring TLS certificates: restoring a snapshot without the matching server.crt and server.key will prevent the cluster from starting.

Limitations

• Embedded etcd is only supported on servers; agent nodes never participate in etcd.
• etcd snapshots are local; for off‑site backups you must copy the snapshot file and TLS certs to a safe location.
• k3s does not support dynamic resizing of the etcd cluster after initial bootstrap.

Bottom Line

Deploying a three‑node k3s cluster with embedded etcd is a matter of bootstrapping the first server, joining two more, and ensuring the required ports are open. Verify health with kubectl and etcdctl, back up snapshots with TLS, and avoid the common mistakes listed above. With these steps, you’ll have a lightweight, highly available Kubernetes cluster that can survive a single node failure.