k3OS SQLite Datastore: Simpler State for Small Clusters

Embedded SQLite removes a moving part

k3OS ships with SQLite compiled directly into its binary. When the control‑plane starts, it opens a single /var/lib/rancher/k3s/server/db/state.db file instead of launching a separate etcd process. This eliminates the need to manage etcd certificates, quorum sizing, and a dedicated data directory on each master node.

For clusters that run fewer than a few hundred pods — typical for edge gateways, CI runners, or developer sandboxes — the SQLite backend satisfies the Kubernetes API latency budget without the operational overhead of a distributed key‑value store.

Backup is a file copy

Because the entire cluster state lives in one SQLite file, a consistent snapshot is as simple as copying that file while the API server is quiesced. The procedure works on any node that holds the control‑plane database (usually the first master).

# Run on the control‑plane node as root
systemctl stop k3s                # quiesce the API server
cp /var/lib/rancher/k3s/server/db/state.db \
   /mnt/backups/k3s-state-$(date +%F).db
systemctl start k3s

Permissions: root (or a user with CAP_DAC_OVERRIDE) is required to stop the service and read the database file. Expected check: after restart, kubectl get nodes should return Ready for all members. Risk: copying while the server writes can corrupt the database; always stop the service first.

HA is not built‑in

SQLite provides no leader election, replication, or automatic failover. If the control‑plane node disappears, the cluster loses its authoritative state until a new master is bootstrapped and the database file is restored manually. This makes SQLite unsuitable for production workloads that demand multi‑master high availability.

Typical symptoms of the limitation appear when you add a second master: the new node will start its own SQLite instance, resulting in split‑brain state. The official documentation therefore recommends SQLite only for single‑master or very small clusters.

Worked example: validate latency on a 100‑pod test cluster

1. Provision three VMs (1 CPU, 2 GiB RAM) and install the same k3OS release that lists “embedded SQLite datastore” in its release notes.
2. Boot the first node as the control plane; join the other two as workers.
3. Deploy a synthetic workload:

   kubectl create deployment load --image=ghcr.io/k3s-io/k3s:latest --replicas=100
   kubectl expose deployment load --port=80 --target-port=8080

4. Measure API latency from a client pod:

   kubectl run curl --rm -i --image=curlimages/curl -- \
     -w "%{time_total}" -o /dev/null -s http://load.default.svc.cluster.local

5. Compare the reported seconds against the same test on an etcd‑backed k3s cluster of identical size. In practice the median latency stays under 30 ms for both backends, confirming the research claim that SQLite meets typical requirements up to ~500 pods.

If the latency spikes or the API server returns 5xx errors, the SQLite file may be growing beyond the node’s storage budget — monitor du -h /var/lib/rancher/k3s/server/db/state.db regularly.

When to stick with etcd

• Clusters that require multiple control‑plane nodes for HA.
• Environments where automated failover and quorum‑based writes are a compliance requirement.
• Workloads that exceed a few hundred pods or generate heavy watch traffic (e.g., large‑scale operators).

Next steps

1. Check the k3OS release notes for your target version and confirm “embedded SQLite datastore” is listed.
2. Spin up a minimal test cluster (as in the worked example) and run the latency comparison.
3. Perform a backup‑restore drill: stop k3s, copy state.db to a safe location, delete the original, copy it back, restart k3s, and verify kubectl get all --all-namespaces matches the pre‑backup state.
4. Document the maximum acceptable database size for your edge nodes and set a monitoring alert on state.db growth.

Following these steps gives you a reproducible baseline before committing SQLite as the default datastore for your fleet.