Managing Cluster State with the K3s Embedded SQLite Datastore

The Trade-off: SQLite vs. etcd for Edge Clusters

Standard Kubernetes requires etcd, a distributed key-value store that ensures consistency across multiple control plane nodes. However, etcd is resource-intensive and requires a quorum of nodes to function. For single-node edge deployments or lightweight development environments, this overhead is often unnecessary.

K3s solves this by using an embedded SQLite database as the default datastore for single-server installations. This replaces the need for a separate etcd process, significantly reducing CPU and memory consumption while maintaining the same Kubernetes API behavior. The primary takeaway is that SQLite is the optimal choice for single-node clusters where resource efficiency is prioritized over high availability (HA).

How the SQLite Datastore Operates

In a default K3s installation, the server process manages the SQLite database internally. All cluster state—including Pod definitions, Secrets, and ConfigMaps—is persisted to a local file on the host filesystem. This removes the network overhead associated with distributed consensus algorithms like Raft used by etcd.

Datastore Location and Persistence

The cluster state is stored in a directory dedicated to the K3s server. On most Linux distributions, this is located at /var/lib/rancher/k3s/server/db/. The core state is held in a file typically named state.db.

Example: Verifying and Backing Up the SQLite State

To confirm your cluster is using SQLite and to perform a manual state backup, follow these steps. These commands must be run on the control plane node with root or sudo permissions.

# 1. Verify the datastore type in the server logs
journalctl -u k3s | grep "using sqlite"

# 2. Check for the existence of the database file
ls -lh /var/lib/rancher/k3s/server/db/

# 3. Create a cluster snapshot (backup)
# This creates a consistent snapshot of the SQLite DB without stopping the server
k3s server snapshot

The k3s server snapshot command is the safe way to back up the state. It ensures the database is not in the middle of a write operation, preventing the corruption that occurs when simply copying the state.db file while the server is running.

Comparison: SQLite vs. External HA Datastores

While SQLite is efficient, it cannot be shared across multiple control plane nodes. If your requirements shift from a single-node setup to a High Availability (HA) cluster, you must migrate to a distributed datastore.

Critical Limitations and Common Pitfalls

Database Locking

SQLite uses file-level locking. In environments with extremely high API write loads (e.g., frequent updates to Custom Resource Definitions or aggressive monitoring probes), you may encounter database is locked errors. This happens when the API server cannot acquire a write lock quickly enough. If this occurs, you should migrate to an external database or etcd.

Manual File Manipulation

Risk: Never attempt to modify the state.db file using an external SQLite browser or CLI while the K3s server is active. Direct modification bypasses the Kubernetes API validation and will likely lead to cluster state corruption, rendering the control plane unbootable.

Migration Path

Moving from SQLite to an HA datastore is not a simple configuration change; it requires a specific installation flag. To use an external database, you must provide the --datastore-endpoint flag during the initial server startup. For example:

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://user:pass@tcp(hostname:3306)/database"

Practical Verification

To ensure your datastore is functioning correctly and is backed up, perform a dry-run recovery check: copy your latest snapshot to a separate directory and attempt to start a temporary K3s instance using that snapshot. If the server boots and kubectl get nodes returns the expected output, your backup chain is verified.