Choosing Between SQLite and etcd for K3s Cluster Storage

The Storage Decision: Stability vs. High Availability

When deploying K3s, the most critical engineering decision is selecting the datastore. Choosing the wrong backend can lead to unnecessary resource consumption on edge devices or catastrophic data loss during a node failure. The primary trade-off is between SQLite (single-node simplicity) and embedded etcd (multi-node resilience).

Comparing Datastore Architectures

K3s simplifies the Kubernetes control plane by bundling the database. Depending on your availability requirements, you will use one of two primary internal mechanisms:

Feature	SQLite (Default)	Embedded etcd
Node Count	Single server node	1, 3, or 5 server nodes
Resource Overhead	Very Low	Moderate (CPU/RAM for consensus)
Fault Tolerance	None (Single point of failure)	High (Quorum-based recovery)
Complexity	Zero configuration	Requires specific node counts for quorum

Scenario 1: Deploying for Edge/IoT (SQLite)

Use SQLite when deploying to a single device (e.g., Raspberry Pi) where RAM is limited. SQLite stores the cluster state in a local file, removing the overhead of a distributed consensus algorithm.

Implementation:
Run the following command on the target node. You must have sudo privileges. This installation defaults to SQLite if no --cluster-init flag is provided.

curl -sfL https://get.k3s.io | sh -

Verification:
Confirm the database is utilizing the local filesystem by checking the K3s data directory:

ls /var/lib/rancher/k3s/server/db/

If a sqlite3.db file is present, the node is operating in single-node mode.

Scenario 2: Deploying for Production HA (Embedded etcd)

Use embedded etcd when you require High Availability (HA). etcd is a distributed key-value store that ensures the cluster state is replicated across multiple server nodes. If one node fails, the cluster remains operational as long as a quorum (majority) of nodes are healthy.

Implementation:
To initialize the first server node with etcd, use the --cluster-init flag. Run this on the first server node:

curl -sfL https://get.k3s.io | sh -s - server --cluster-init

To join additional server nodes to the cluster, you will need the node token from the first server (located at /var/lib/rancher/k3s/server/node-token) and the IP of the first node:

curl -sfL https://get.k3s.io | sh -s - server --server https://<FIRST_NODE_IP>:6443 --token <NODE_TOKEN>

Risk Warning: Do not deploy an even number of etcd nodes (e.g., 2 or 4). This can lead to "split-brain" scenarios where the cluster cannot reach a consensus on the state, causing the API server to become read-only or crash.

Diagnostic Checks and Validation

Regardless of the storage choice, verify the cluster health using kubectl. This tool interacts with the API server to ensure the datastore is responding correctly.

1. Check Node Status: Run kubectl get nodes -o wide. Ensure all nodes show a status of Ready.
2. Verify Version: Run k3s --version to ensure you are on a supported release (e.g., v1.26+).
3. Resource Monitoring: On devices with less than 256MB of available RAM, monitor for OOMKilled (Out of Memory) events, as the control plane may struggle to schedule pods.

Rollback and State Recovery

Because these operations change the system state and write to disk, recovery depends on the backend:

• SQLite: To reset, stop the service and delete the directory /var/lib/rancher/k3s/server/db/. This wipes all cluster configuration and deployed workloads.
• etcd: Recovery requires etcd snapshots. If the quorum is lost, you must restore from a snapshot using the k3s server --cluster-reset command on a single node before restarting the rest of the cluster.