Run a Highly Available K3s Control Plane with the Embedded SQLite Datastore

Problem: HA Control Plane Without External etcd

Operating a highly available Kubernetes control plane traditionally requires an external etcd cluster, adding operational overhead, extra machines, and network complexity. K3s v1.24+ includes a built-in SQLite datastore (sqlitedb) that uses HashiCorp Raft for consensus, allowing three or more server nodes to form an HA control plane with no external dependencies. This guide walks through provisioning, verifying, and recovering a sqlitedb-backed K3s cluster.

Prerequisites

• Three or more Linux nodes (amd64 or arm64) with a supported OS (Ubuntu 20.04+, Debian 11+, RHEL 8+, etc.).
• Network connectivity between all server nodes on:
  • Port 6443 (Kubernetes API)
  • Port 9001 (SQLite Raft replication)
• Identical K3s version on every node (v1.24 or later).
• A shared secret token (any string) for cluster join authentication.
• Low latency between nodes (<5 ms RTT). Cross‑region HA with sqlitedb is not supported.
• Synchronized clocks (NTP) on all nodes to avoid certificate rotation issues.

Procedure: Bootstrap the Cluster

1. Initialize the First Server

Run the install script with --cluster-init to create the Raft cluster and generate the initial SQLite database.

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --token=CLUSTER_SECRET_TOKEN

Where to run: On the first server node (e.g., 10.0.0.1).
Permissions: Root or sudo.
Risk: Using --cluster-init on more than one node will create independent clusters that cannot merge.

2. Join Additional Servers

On each remaining server node, run the same install script pointing at the first server’s API endpoint. Omit --cluster-init.

curl -sfL https://get.k3s.io | sh -s - server \
  --server=https://10.0.0.1:6443 \
  --token=CLUSTER_SECRET_TOKEN

Where to run: On each additional server node.
Expected behavior: The new node contacts the leader, receives the SQLite snapshot, and starts replicating via Raft on port 9001.

Verification: Confirm HA Operation

Node Readiness

k3s kubectl get nodes -o wide

All server nodes should show STATUS Ready and their internal IPs.

Control‑Plane Endpoints

k3s kubectl get endpoints -n kube-system kube-controller-manager

Output must list multiple IPs (one per server), proving the leader election works across nodes.

SQLite Replication Health

k3s kubectl logs -n kube-system -l component=kube-apiserver --tail=50 | grep -i sqlitedb

Look for lines like replication commit index=123 indicating active Raft log replication.

Snapshot Creation

ls -la /var/lib/rancher/k3s/server/db/snapshots/

Recent .db files (default interval 10 minutes, configurable via --snapshot-interval-seconds) confirm automated backups.

API Server Reachability

for ip in 10.0.0.1 10.0.0.2 10.0.0.3; do
  curl -k https://$ip:6443/healthz
  echo " $ip"
done

Each request should return ok.

Recovery Scenarios

Single Server Failure

1. Provision a replacement node with the same OS and network access.
2. Join it using the same --server and --token as step 2.
3. The new node will catch up via Raft snapshot transfer automatically.

Quorum Loss (Majority Down)

If two of three servers are lost, the remaining node cannot elect a leader. Restore from the latest snapshot:

1. On the surviving node, stop K3s: systemctl stop k3s.
2. Run the reset with restore:

   k3s server \
         --cluster-reset \
         --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/.db

3. Start K3s: systemctl start k3s.
4. Re‑join the other replacement nodes using the standard join command.

Risk: --cluster-reset destroys the existing Raft state on that node; ensure you have the correct snapshot file before proceeding.

Operational Limits and Migration Path

Aspect	sqlitedb Limit	Recommendation
Cluster size	≤100 nodes	Use external etcd for larger clusters
Write throughput	Moderate	High‑churn workloads should use etcd
Network latency	<5 ms	Co‑locate servers in same AZ
Upgrades	Rolling, one server at a time	Never upgrade all simultaneously

Migrating to External etcd

If you outgrow sqlitedb, you can migrate without data loss:

1. Take an etcd backup from a new etcd cluster (or use etcdctl snapshot save).
2. On one K3s server, run:

   k3s server \
         --cluster-reset \
         --cluster-reset-restore-path=/path/to/etcd-snapshot.db

3. Re‑join all servers with --datastore-endpoint=https://etcd1:2379,https://etcd2:2379,https://etcd3:2379.

Quick Diagnostic Checklist

• k3s kubectl get nodes -o wide → all Ready
• k3s kubectl get endpoints -n kube-system kube-controller-manager → multiple IPs
• grep -i sqlitedb in apiserver logs → replication commits present
• ls /var/lib/rancher/k3s/server/db/snapshots/ → recent snapshots
• curl -k https://:6443/healthz → all return ok

Summary

The embedded SQLite datastore gives you a fully functional HA control plane with three commands and no external etcd. Keep latency low, monitor snapshot freshness, and upgrade one server at a time. When the cluster grows beyond ~100 nodes or write load spikes, migrate to etcd using the built‑in reset/restore mechanism.