Choosing Embedded etcd vs External Datastore for Production K3s HA Clusters

The Decision: Embedded etcd or External Datastore?

K3s v1.19+ introduced embedded etcd for highly available control planes, eliminating the need for an external MySQL or PostgreSQL database. For most production teams, this reduces operational complexity — fewer moving parts, integrated snapshots, and automatic bootstrap. But external datastores still make sense when you have existing DBA workflows, need to share a datastore across clusters, or require specific compliance tooling. This guide walks through deploying a production-grade K3s cluster with embedded etcd, verifying it works, and recovering from common failure scenarios.

Desired Outcome

A three-server K3s cluster (v1.28+) with embedded etcd providing HA control plane, all server nodes showing Ready, etcd quorum healthy, and automated snapshots stored off-node for disaster recovery.

Prerequisites

• Nodes: 3+ Linux servers (amd64 or arm64), each with 2+ CPUs, 4 GB+ RAM, unique hostnames, static IPs
• Network: Firewall/open security groups for TCP 6443 (Kubernetes API), TCP 2379-2380 (etcd client/peer), UDP 8472 (Flannel VXLAN), TCP 10250 (kubelet)
• Access: SSH to each node as a user with sudo; systemd init system
• Time sync: chrony or ntp running on all nodes (etcd is sensitive to clock drift)
• SELinux/AppArmor: Permissive or properly configured for container runtimes
• Token: A strong shared secret (32+ random chars) for cluster join; store securely
• TLS SAN: Load balancer IP or FQDN for --tls-san if you will front the API with a load balancer

Cluster Initialization Procedure

1. Bootstrap the first server node

Run on the first server (replace placeholders):

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --token=<shared-secret> \
  --tls-san=<lb-ip-or-fqdn>

Where: First server node via SSH. Permissions: sudo (install script uses systemd). Placeholders: <shared-secret> = your generated token; <lb-ip-or-fqdn> = load balancer VIP or DNS name (omit if no LB). Risk: This command installs K3s as a system service and writes to /var/lib/rancher/k3s. Ensure no existing Kubernetes components conflict.

2. Join additional server nodes

On each of the remaining two (or more) server nodes:

curl -sfL https://get.k3s.io | sh -s - server \
  --server=https://<first-node-ip>:6443 \
  --token=<shared-secret>

Where: Each additional server node. Note: Do not use --cluster-init on joiners. Version skew: All server nodes must run the exact same K3s version; the install script pulls latest by default — pin with INSTALL_K3S_VERSION=v1.28.12+k3s1 if needed.

3. Add agent (worker) nodes

On each worker node:

curl -sfL https://get.k3s.io | sh -s - agent \
  --server=https://<first-node-ip>:6443 \
  --token=<shared-secret>

Expected Verification Checks

Run these from any server node (or a workstation with kubectl pointed at the cluster):

Node and control-plane health

kubectl get nodes -o wide

All server and agent nodes should show Ready and the correct internal IPs.

etcd pods running on each server

kubectl get pods -n kube-system -l component=etcd -o wide

Expect one etcd-<node> pod per server node, all Running.

Control-plane components distributed

kubectl get pods -n kube-system -l tier=control-plane -o wide

kube-apiserver, kube-controller-manager, kube-scheduler pods should be spread across the three servers.

API endpoint reflects all servers

kubectl get endpoints kubernetes -o yaml

The subsets.addresses list should contain all three server IPs on port 6443.

CNI and DNS healthy

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get pods -n kube-system -l app=flannel

CoreDNS pods Ready; Flannel (or your chosen CNI) pods Running on every node.

etcd cluster health (direct check)

k3s kubectl exec -n kube-system etcd-$(hostname) -- \
  etcdctl endpoint health --cluster \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key

Each endpoint should report healthy. Run from a server node; requires k3s CLI (installed with server).

Snapshot Backup and Off-Node Storage

Embedded etcd snapshots are written locally to /var/lib/rancher/k3s/server/db/snapshots/. For disaster recovery, copy them off-node.

Manual snapshot

k3s etcd-snapshot save --name=manual-$(date +%s)

Confirm file appears in the snapshots directory.

Automated off-node copy (cron example)

On each server, add a cron entry to push snapshots to an NFS mount or S3-compatible store:

0 2 * * * root k3s etcd-snapshot save --name=auto-$(date +\%Y\%m\%d) && \
  rsync -av /var/lib/rancher/k3s/server/db/snapshots/ nfs-host:/backups/k3s/$(hostname)/

Limitation: Snapshots are node-local by default; K3s does not replicate them across servers. If the only server with a recent snapshot fails, you lose that backup.

Recovery Scenarios

Single server failure (quorum maintained)

Two of three servers remain — etcd quorum holds. The failed node shows NotReady after ~5 minutes. To replace:

1. Provision a new node with same hostname/IP (or update DNS/LB).
2. Run the join command from step 2 with the same --token.
3. Verify it appears Ready and etcd pod starts.

No snapshot restore needed; the new node catches up via etcd replication.

Quorum loss (2/3 servers down)

Cluster API becomes unavailable. Recovery requires restoring from snapshot on a surviving server:

1. Stop K3s on the surviving server: systemctl stop k3s
2. Run restore: k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<snapshot-file> --token=<shared-secret>
3. Start K3s: systemctl start k3s
4. Rejoin the other server nodes using the standard join command.

Critical: The K3s version on the restore node must exactly match the version that created the snapshot (v1.24+ snapshot format is version-sensitive).

Full cluster loss

1. Provision three new servers meeting prerequisites.
2. On the first, run k3s server --cluster-init --cluster-reset --cluster-reset-restore-path=<snapshot> --token=<shared-secret> --tls-san=<lb-ip>
3. Join remaining servers and agents normally.

Certificate rotation

K3s rotates certificates automatically. To force immediate rotation on all servers:

systemctl restart k3s

Run sequentially (one server at a time) to avoid brief API disruption.

Embedded etcd vs External Datastore: Practical Comparison

Factor	Embedded etcd (HA)	External MySQL/PostgreSQL
Operational components	K3s only	K3s + managed DB + networking
Bootstrap/join	Automatic via --cluster-init	Manual DB schema init, connection strings
Backups	Integrated etcd-snapshot CLI	DBA tooling (mysqldump, pg_basebackup)
Network latency	Local etcd, peer on 2380	Extra hop to DB; split-brain risk if multi-AZ
Upgrade complexity	K3s version only	K3s + DB engine version coordination
Team expertise required	Kubernetes	Kubernetes + DBA
Compliance/audit	etcd audit logs	DB audit logs (often richer)

Default recommendation: Use embedded etcd unless you have a hard requirement for external datastore (existing multi-cluster DB, regulatory mandate for specific DB audit features, or team lacks Kubernetes ops but has strong DBA staff).

Limitations and Gotchas

• No migration path: Cannot switch between embedded etcd and external datastore without full cluster rebuild.
• Quorum math: 3 servers tolerate 1 failure; 5 tolerate 2. Even-numbered server counts waste a node (4 servers still tolerate only 1).
• ARM64 performance: etcd on ARM64 can exhibit higher latency under heavy write load; benchmark with your workload before committing.
• Flannel VXLAN (UDP 8472): Cloud security groups often block this by default — verify cross-node pod connectivity.
• Token rotation: Default token lives in /var/lib/rancher/k3s/server/node-token. If compromised, rebuild cluster with new token.

Verification Checklist Before Production

1. All three servers Ready and etcd pods Running
2. etcdctl endpoint health --cluster returns healthy for all three
3. Snapshot saves and appears in /var/lib/rancher/k3s/server/db/snapshots/
4. Off-node copy of snapshot succeeds (test restore on spare node)
5. Simulated single-server power-off: cluster remains available, node rejoins automatically on power-on
6. kubectl auth can-i '*' '*' --all-namespaces returns yes for cluster-admin