Choosing a Datastore for K3s HA: SQLite, Embedded etcd, or External SQL?

The problem: one cluster, three datastore paths

K3s makes it trivial to spin up a single-node cluster — it defaults to SQLite, a single file under /var/lib/rancher/k3s/server/db. That works fine until you need high availability. The moment you add a second server node, SQLite is no longer an option; sharing the file over NFS leads to corruption and split-brain. You must pick between embedded etcd (the Kubernetes default) or an external SQL database (PostgreSQL, MySQL, MariaDB). Each choice locks you into different operational models, backup procedures, and migration constraints.

Embedded etcd: strong consistency, operational weight

Embedded etcd gives you the same consistency guarantees as upstream Kubernetes. You need an odd number of server nodes (3, 5, 7) to form a quorum. K3s handles leader election and certificate rotation automatically (since v1.24). The trade-off: each server runs an etcd member consuming 200–400 MB RAM and demanding low-latency LAN (<5 ms RTT). Disk I/O matters — put etcd on dedicated SSDs if possible.

Backups use k3s etcd-snapshot (automated via cron). Restore is a cluster reset operation: k3s server --cluster-reset --cluster-reset-restore-path=<snapshot-dir> on a fresh node. Migration from SQLite to embedded etcd is a one-way, offline process: start the first server with --cluster-init, then join additional servers with the generated token.

External SQL: stateless servers, managed database

With --datastore-endpoint='postgres://user:pass@host/db', K3s servers become stateless SQL clients. HA shifts to the database layer — you can use RDS, Cloud SQL, or a self-managed PostgreSQL cluster with synchronous replication. Server nodes only need ~5 MB for the SQL client library. Network latency tolerance is higher, but you must configure TLS and connection pooling (PgBouncer, ProxySQL) to avoid connection storms during control-plane restarts.

Backup and restore move to native tooling: pg_dump, cloud snapshots, or point-in-time recovery. Major K3s upgrades may require manual schema migrations — check release notes. A critical limitation: external SQL mode does not support Kubernetes API server features that rely on etcd watch semantics (certain CRD conversion webhooks). Verify compatibility before committing.

Worked example: bootstrapping a 3-node HA cluster with PostgreSQL

Assume a managed PostgreSQL instance (multi-AZ, synchronous replication) at pg-cluster.example.com:5432 with database k3s, user k3s, password in /run/secrets/pg_password.

1. On the first server node, create the datastore endpoint string:

   export DATASTORE="postgres://k3s:$(cat /run/secrets/pg_password)@pg-cluster.example.com:5432/k3s?sslmode=verify-full&sslrootcert=/etc/ssl/certs/ca-certificates.crt"

2. Start the first server (creates schema):

   k3s server \
     --datastore-endpoint="$DATASTORE" \
     --tls-san pg-cluster.example.com \
     --node-label "role=server"

3. Grab the join token from /var/lib/rancher/k3s/server/node-token.
4. On each additional server node, join with the same datastore endpoint:

   k3s server \
     --datastore-endpoint="$DATASTORE" \
     --token "$(cat /path/to/node-token)" \
     --tls-san pg-cluster.example.com \
     --node-label "role=server"

5. Verify all servers report Ready:

   kubectl get nodes -o wide

Agent nodes join normally with k3s agent --server https://<server-ip>:6443 --token <token> — they don't need the datastore endpoint.

Trade-off: migration is a one-way door

You cannot live-migrate between datastore backends. SQLite → embedded etcd requires cluster rebuild with --cluster-init. SQLite or etcd → external SQL requires a fresh cluster and workload redeployment. Plan your datastore choice before production workloads land. If you start with SQLite on a single node and later need HA, you'll rebuild.

Actionable checklist

• Single node, no HA needed? Stay with SQLite — simplest backups (file copy), zero dependencies.
• Need HA, have 3+ servers on low-latency LAN, want Kubernetes-native semantics? Use embedded etcd.
• Prefer managed database, have PostgreSQL/MySQL expertise, accept watch-semantics limitations? Use external SQL with connection pooling and multi-AZ failover tested.
• Run k3s server --help | grep -A2 -B2 'datastore\|etcd\|sql' to confirm supported drivers for your version.
• Test restore procedures before you need them — snapshot/restore for etcd, pg_dump/cloud snapshot for external SQL.