Diagnosing RocksDB Write Stalls: From 'Stalling Writes' to a Targeted Fix

Your RocksDB-backed service suddenly stops accepting writes for seconds at a time, latency graphs show cliffs instead of curves, and the LOG file fills with lines like Stalling writes because of estimated pending compaction bytes or Stopping writes because we have too many level-0 files. That is a write stall: RocksDB deliberately throttles or blocks writers because background work (flush and compaction) is not keeping up with the write rate. The fix is almost never "turn off the stall" — it is finding which of three backlogs triggered it and fixing that one.

What the stall messages are telling you

RocksDB protects itself with configurable triggers. When a backlog crosses a slowdown threshold, writes are delayed; when it crosses a stop threshold, writes block entirely. There are three independent backlogs, and each maps to a different root cause:

LOG message / symptom	Backlog	Likely cause
"too many memtables"	Immutable memtables waiting to flush	Flush too slow: too few flush threads, slow device, or memtables filling faster than they drain
"too many level-0 files"	L0 SST file count vs level0_slowdown_writes_trigger / level0_stop_writes_trigger	L0→L1 compaction not keeping up; often a parallelism or per-level size problem
"estimated pending compaction bytes"	Total bytes queued for compaction vs soft/hard pending-compaction limits	Deep compaction backlog in lower levels; sustained write rate exceeds compaction throughput

Option names and default trigger values vary across RocksDB versions, so read the actual values from your LOG header or installed headers rather than assuming documented defaults.

Ordered checks to run during a stall

Run these in order; each one narrows the cause. All property reads happen in your application process where the DB* handle lives — no special permissions needed beyond what the process already has.

1. Grep the LOG for stall reasons. Search for Stalling and Stopping. The message names the backlog directly, which usually answers the "which one" question immediately.
2. Check for background errors. Read the property rocksdb.background-errors. A non-zero value means a flush or compaction failed (disk full, corruption, I/O error) and tuning will not help until that is resolved.
3. Snapshot live state. Query these properties via DB::GetProperty (names may differ slightly by version):

   rocksdb.num-immutable-mem-table   // memtables queued for flush
   rocksdb.num-files-at-level0       // L0 file count
   rocksdb.estimate-pending-compaction-bytes
   rocksdb.mem-table-flush-pending
   rocksdb.compaction-pending

   Also dump rocksdb.cfstats (includes a STALL section with cumulative stall time per cause) and rocksdb.levelstats (per-level file counts and compaction score).
4. Check the device. On the host, run iostat -x 5 (Linux, requires the sysstat package, no root needed for your own disks). If %util is pinned near 100 and await is high during stalls, the storage device itself may be the ceiling — no RocksDB option fixes that.

Distinguishing flush-bound from compaction-bound

The diagnostic fork that matters most: if num-immutable-mem-table is high but L0 file count is low, you are flush-bound — memtables cannot drain to disk fast enough. If L0 files or pending compaction bytes are high while immutable memtables stay low, you are compaction-bound — data reaches L0 fine but cannot move downward fast enough. Treating a flush-bound system with compaction tuning (or vice versa) wastes time and can make stalls worse.

Fixes tied to findings

If flush-bound

• Increase background flush capacity: raise max_background_jobs (modern versions pool flush and compaction threads; older versions use separate max_background_flushes).
• Give flush more breathing room: raise max_write_buffer_number or write_buffer_size so more memtables can accumulate before the stall triggers. Watch total memory: the global cap db_write_buffer_size governs the memtable budget across column families, and larger buffers also mean more WAL to replay after a crash, lengthening recovery.
• If the device is the bottleneck (per iostat), no option helps — see escalation below.

If compaction-bound at L0

• Increase compaction parallelism via max_background_jobs and, on versions that support it, subcompactions.
• Tune level0_file_num_compaction_trigger: a lower value starts L0→L1 compaction earlier (smoother, more write amplification); a higher value tolerates bursts but makes each compaction bigger.
• Resist simply raising the L0 slowdown/stop triggers. That masks the backlog and degrades read latency, because every point lookup may have to probe more L0 files.

If pending compaction bytes are the trigger

• For leveled compaction, check that max_bytes_for_level_base and target_file_size_base match your data volume; an undersized L1 forces constant re-compaction.
• Consider whether the compaction style fits the workload. Universal compaction tolerates write bursts with lower write amplification but needs more space and can hurt read performance; leveled compaction gives tighter read and space amplification at higher write cost. Switching styles is a major decision — test it under representative load, not in production.
• Enable a rate limiter (rate_limiter option) to smooth flush/compaction I/O bursts so foreground writes get steadier device access.

Verifying a fix

Reproduce the stall under controlled load before changing anything, so you have a baseline. db_bench (built from the RocksDB source tree, run on a test host or against a scratch DB path — never against production data) works well:

./db_bench --db=/scratch/bench --num=10000000 \
  --benchmarks=fillrandom --compression_type=snappy

Confirm the stall counters appear in the LOG, apply one change, rerun, and compare: total stall time in the rocksdb.cfstats STALL section, p99 write latency, and the live properties above during the run. A fix that does not reduce stall time for the identified cause means the diagnosis was wrong — go back to the checks.

When to escalate

Stop tuning and escalate when: stalls persist after flush and compaction parallelism are sized to the hardware; rocksdb.background-errors is non-zero (investigate the underlying I/O or corruption first); iostat shows the device saturated even with rate limiting (a storage/hardware review is needed); or write amplification computed from rocksdb.cfstats is extreme for the workload — at that point the data model, key design, or sharding strategy, not RocksDB options, is the right lever.