k3s and SQLite: API Server Latency During Concurrent Write Operations

Answer to the Core Question

In k3s v1.28+ with the default SQLite backend via kine, the API server’s write latency starts to be dominated by SQLite contention when:

• Concurrent write requests exceed roughly 10–20 simultaneous operations on a typical SSD‑backed node.
• The aggregate write rate climbs above 50 writes per second (WPS), or the workload demands sub‑10 ms latency for object updates.

When either of these thresholds is crossed, the API server will begin queuing writes, emit “database is locked” errors, or return 504/timeout responses. At that point, moving to an external PostgreSQL or MySQL datastore is recommended to keep response times stable.

Confirmed Facts vs. Likely Explanations

Confirmed facts (based on k3s and Kine documentation + community benchmarks):

• SQLite in WAL mode still allows only one writer at a time; write contention is a single‑writer lock.
• Standard SSDs can clear a write lock in ~2–5 ms; exceeding ~10–20 concurrent writes saturates this window.
• k3s logs surface “database is locked” or context deadline exceeded when contention is high.

Likely explanations (needs verification in your environment):

• The exact contention point depends on the underlying filesystem and storage medium (NVMe vs. SATA SSD vs. NFS).
• Network‑attached storage dramatically raises I/O latency, lowering the safe concurrent write count.

Practical Steps to Verify Your Cluster

1. Measure write throughput: Run a lightweight load test (e.g., kubectl create -f resource.yaml --record) repeatedly while monitoring k3s-server logs for “database is locked”.
2. Track API latency: Use kubectl top pod k3s-server or Prometheus metrics for apiserver_request_latency_seconds to see when latency spikes.
3. Calculate WPS: Count the number of POST/PUT/PATCH requests per second using kubectl logs -f k3s-server | grep -c "POST" or a dedicated metrics endpoint.
4. Compare the observed WPS and latency against the 10–20 concurrent write and 50 WPS thresholds above.
5. If you hit or exceed those numbers, consider migrating to PostgreSQL or MySQL. The migration involves adding the --datastore-endpoint flag and deploying a compatible Kine backend.

Missing Diagnostic Detail

To fine‑tune the recommendation, could you share the type of storage backing the SQLite file on your k3s nodes (local SSD, NVMe, or NFS)? This influences the exact contention point.