Choosing Embedded etcd in K3s: When, How, and What to Watch

Problem Statement

Running a K3s cluster on multiple servers raises the question of how the control‑plane data is stored. The default SQLite backend works for single‑node or highly constrained edge devices, but it cannot be replicated across nodes. When you need high‑availability (HA) for the API server and controller manager, you must move to a replicated datastore. K3s offers an “embedded etcd” option that runs a lightweight etcd instance inside each server process. This article explains when to choose it, how to configure it, and what operational trade‑offs you should consider.

Why Embedded etcd Matters

etcd is the key-value store used by Kubernetes to persist cluster state. In a HA K3s deployment, each server node runs an etcd member that participates in a quorum. If the quorum is lost, the cluster becomes read‑only or unavailable. SQLite, by contrast, is a single file stored on the local node; it cannot survive a node failure. Therefore, for any production‑grade multi‑server cluster you should use embedded etcd.

Key points:

• Embedded etcd requires an odd number of server nodes (3, 5, …) to maintain quorum.
• A cluster token authenticates new server nodes and ensures they join the same etcd cluster.
• etcd snapshots are automatically created and can be used for disaster recovery.
• Memory and disk usage increase compared to SQLite – be mindful on resource‑constrained hosts.

Setting Up a 3‑Node HA Cluster

Below is a minimal example that demonstrates how to bootstrap a 3‑node K3s cluster with embedded etcd. Replace 10.0.0.1, 10.0.0.2, and 10.0.0.3 with your actual server IPs.

# On the first node (leader)
# Install K3s with embedded etcd and specify a cluster token
sudo k3s server \
  --cluster-init \
  --datastore-endpoint="" \
  --node-taint "CriticalAddonsOnly=true:NoExecute" \
  --token <cluster-token>

# On the second node
sudo k3s server \
  --server https://10.0.0.1:6443 \
  --token <cluster-token>

# On the third node
sudo k3s server \
  --server https://10.0.0.1:6443 \
  --token <cluster-token>

After the third node joins, run kubectl get nodes from any node to confirm all three are Ready. The k3s server process will launch an embedded etcd instance listening on ports 2379/2380 by default.

Verifying the Datastore Backend

On any server node, inspect the data directory (default /var/lib/rancher/k3s/server/db).

# List contents
ls -l /var/lib/rancher/k3s/server/db
# Look for the etcd snapshot directory
ls -l /var/lib/rancher/k3s/server/db/etcd

The presence of a etcd subdirectory and snapshot files confirms that embedded etcd is active. If you see only a SQLite file (k3s.db), the cluster is still using SQLite.

Checking Running Processes

Embedded etcd runs as a separate process inside the K3s server. Verify it with:

ps -ef | grep etcd | grep k3s
# Example output (not guaranteed):
# root    1234  1  0 09:21 ?        00:00:02 /usr/local/bin/k3s server --datastore-endpoint="" ...
# root    1235  1234 0 09:21 ?        00:00:00 /usr/local/bin/k3s server ... etcd

Managing Snapshots

etcd snapshots are created automatically in the etcd directory. You can adjust the retention policy with the --etcd-snapshot-retention flag (default 5 snapshots). To trigger a manual snapshot:

sudo k3s server --snapshot --snapshot-dir /var/lib/rancher/k3s/server/db/etcd

Restoring from a snapshot requires stopping all server nodes, replacing the entire db directory on each node with the snapshot contents, and restarting the cluster. This is a disruptive operation, so plan for maintenance windows.

Trade‑offs & Limitations

• Resource Footprint: Embedded etcd consumes roughly 200 MiB of RAM per node and writes to the local disk. On lightweight edge devices, this overhead may be prohibitive.
• Migration Path: Switching from SQLite to embedded etcd after the cluster is created is not supported. You must create a fresh cluster and migrate workloads.
• Quorum Requirements: With an even number of server nodes, etcd cannot form a quorum. Plan for 3, 5, or 7 nodes for HA.
• Snapshot Restore Complexity: Full cluster restore requires coordinated downtime across all nodes. Automating this with scripts or CI/CD can mitigate manual errors.

Practical Next Steps

1. Decide on the datastore early in the design phase. If you anticipate scaling beyond a single node or need true HA, opt for embedded etcd.

2. Generate a secure cluster token (e.g., openssl rand -hex 32) and keep it secret.

3. Monitor etcd health with kubectl get componentstatuses and check logs for etcd failures.

4. Plan snapshot retention and backup strategy. Store snapshots off‑site or in object storage if possible.

5. Test failover by taking down one server node and verifying that the cluster remains functional.

By following these guidelines, you can confidently deploy a K3s cluster that is resilient, backed by a replicated datastore, and ready for production workloads.

Diagram

Below is a conceptual diagram of the three‑node embedded etcd setup.

Node	Role	etcd Member
Server 1	Leader	etcd Member
Server 2	Follower	etcd Member
Server 3	Follower	etcd Member