Diagnosing SQLite‑Based etcd in K3s Clusters: A Step‑by‑Step Diagnostic Guide

Recognizable condition

After a node reboot or a k3s upgrade, the API server becomes unreachable. `kubectl` commands return “connection refused” or “etcd server is unavailable”, and pod statuses stay in Pending. The cluster appears healthy in the UI, but the control plane cannot serve API requests.

Diagnostic snapshot

Symptom	Likely cause	Quick check
API server down, `kubectl` errors	SQLite etcd datastore corrupted or not started	`k3s kubectl get componentstatuses` shows etcd unhealthy
Pods stuck in Pending	etcd cannot locate cluster state	`journalctl -u k3s-server` contains “etcd: database path not found”
Node restarts cause state loss	SQLite file lost or moved without migration	File path in `ps aux | grep etcd` differs between nodes

Ordered checks

1. Verify component health
   Run on the control‑plane node (the one running `k3s-server`):

   sudo k3s kubectl get componentstatuses

   All components should be Healthy. If etcd shows Unhealthy or missing, proceed to the next step.

2. Inspect server logs for etcd startup
   

   sudo journalctl -u k3s-server -n 20 --no-pager

   Look for a line similar to “etcd started with data directory /var/lib/rancher/k3s/server/db”. If the path points to a non‑existent location, the datastore is mis‑configured.

3. Confirm the SQLite file location on each node
   

   ps aux | grep etcd

   Every node in a SQLite‑only cluster must list the same path, e.g. /var/lib/rancher/k3s/server/db/etcd.db. If a node shows a different path or an external etcd address, the cluster is mixing storage modes, which is unsupported.

4. Check write contention
   If your workload performs many small writes (e.g., frequent pod creation/deletion), run:
   

   sudo k3s kubectl get nodes -o wide

   High write rates (>100 writes/sec) can cause SQLite file‑locking delays. Observe API latency with:
   

   sudo k3s kubectl get --raw "/api/v1/nodes"

   A sudden rise in response time suggests the embedded SQLite etcd is a bottleneck.

Fixes tied to findings

• Correct datastore path
  If the path in the log or `ps` output does not match /var/lib/rancher/k3s/server/db/etcd.db, edit the k3s service file to point to the right directory, then restart:
  

  sudo systemctl restart k3s-server

  Risk: a wrong path will prevent etcd from starting, taking the whole cluster offline. Verify with `k3s kubectl get componentstatuses` after restart.

• Restore from backup
  When the SQLite file is corrupted, recover using a recent backup:
  

  sudo cp /backup/etcd-2026-11-01.db /var/lib/rancher/k3s/server/db/etcd.db
  sudo systemctl restart k3s-server

  After restart, run `k3s kubectl get nodes` to confirm API availability. Risk: restoring an outdated backup may lose recent state; ensure the backup is no older than the last successful `kubectl get pods` output.

• Migrate to external etcd
  For clusters exceeding three nodes or requiring high availability, switch to external etcd:
  

  sudo k3s install --datastore-endpoint=https://external-etcd:2379

  This command starts k3s with the external endpoint and automatically migrates the SQLite file to the new datastore. Verify with the same `componentstatuses` check. Risk: downtime during migration; schedule a maintenance window.

• Upgrade k3s version
  If you are on a k3s release older than v1.27, upgrading to a newer version may change etcd behavior (e.g., stricter file‑locking). Review the release notes, then perform a rolling upgrade on each node, confirming health after each step.

Escalation criteria

• The SQLite file is missing after a node reboot, and no backup exists – the cluster state is lost. Escalate to a full cluster recreation or restore from a snapshot taken before the incident.
• Persistent “etcd server is unavailable” errors despite a healthy SQLite file indicate a deeper storage‑layer issue (e.g., filesystem corruption). Escalate to the underlying OS support team and consider moving to external etcd.
• Write contention causing API latency above 500 ms for more than 10 minutes suggests the embedded SQLite etcd cannot scale. Escalate by designing a dedicated external etcd cluster or increasing node count to distribute load.

Practical verification after fixes

1. Run `sudo k3s kubectl get componentstatuses`; all components must be Healthy.
2. Execute `sudo journalctl -u k3s-server -n 5 --no-pager` and confirm etcd startup messages reference the correct SQLite path.
3. Perform a quick API test: `sudo k3s kubectl run test-pod --image=busybox --restart=Never -- sleep 30`; then `sudo k3s kubectl get pods` should show the pod running.
4. If you migrated to external etcd, verify that `ps aux | grep etcd` on each node shows the external endpoint address rather than a local file path.

By following this diagnostic guide you can quickly identify whether the embedded SQLite etcd is the root cause of API failures, apply the appropriate fix, and know when to move beyond the built‑in datastore for a more resilient K3s deployment.