RocksDB Write Buffer Size and Max Write Buffer Number Interaction Causing Write Stalls

Question

Goal: Identify a safe combination of write_buffer_size and max_write_buffer_number that avoids write stalls or out‑of‑memory kills on production nodes with limited RAM while maintaining ingest throughput.

Constraints: Production machines may have less memory than developer workstations, max_background_compactions can be limited to reduce CPU usage, and the stall trigger depends not only on write_buffer_size * max_write_buffer_number but also on the current level‑0 file count and the soft_pending_compaction_bytes threshold.

Uncertainty: The exact point at which RocksDB switches from allowing new memtables to triggering a write stall is not exposed directly, making it hard to predict whether a given configuration will stall under sustained write load.

What is the effective memtable memory threshold that triggers a write stall when level‑0 files exceed a certain count?
How should max_background_compactions be proportioned to write_buffer_size * max_write_buffer_number to prevent stalls under a given write rate?
Is there a practical guideline for adjusting write_buffer_size and max_write_buffer_number based on available RAM and expected compaction throughput?

Tasadduq Burney · Answer

In RocksDB, write stalls occur when the total memory consumed by active memtables (defined by `write_buffer_size * max_write_buffer_number`) reaches its limit, and the database cannot flush these buffers to disk fast enough. To avoid stalls without triggering Out-of-Memory (OOM) kills, you must balance ingest bursts against compaction throughput.

The Stall Mechanism
There is no single 'memory threshold' that triggers a stall; rather, a stall is triggered by one of two conditions:

Memtable Exhaustion: The number of memtables reaches `max_write_buffer_number`, and all existing memtables are currently being to disk.
  Level-0 Overload: The number of files in Level-0 exceeds `max_level_0_file_num`. RocksDB stops new writes until compaction moves these files to Level-1.

Proportioning Compaction
To prevent stalls under sustained load, `max_background_compactions` must be sufficient to clear Level-0 files and merge SST files faster than they are being created. If your write rate is high, a low compaction count will cause Level-0 to pile up, triggering a stall regardless of how much RAM you have allocated to write buffers.

Practical Guidelines for Limited RAM
When running on memory-constrained-production nodes, follow these steps to find a safe configuration:

Calculate Memory Ceiling: Ensure `(write_buffer_size * max_write_buffer_number)` does not exceed 50-60% of available system RAM to leave room for the block cache and overhead.
  Prioritize Count over Size: Use a smaller `write_buffer_size` (e.g., 64MB-128MB) with a higher `max_write_buffer_number` (e.g., 4 to 8). This allows RocksDB to handle spikes in traffic without requiring massive contiguous memory blocks.
  Adjust Level-0 Limits: If stalls persist despite low memtable usage, increase `max_level_0_file_num` (default is 4) to give the background compactor more breathing room before a hard stall is enforced.
  Align Compaction: Set `max_background_compactions` to at least 2-4, provided CPU cycles allow, to ensure the pipeline can keep up with the ingest.

Assumption: This analysis assumes you are using the default LSMTable implementation. If using the Column Family, the memory management and stall logic differ significantly based on block-based allocation.

RocksDB Write Buffer Size and Max Write Buffer Number Interaction Causing Write Stalls

1 answer

The Stall Mechanism

Proportioning Compaction

Practical Guidelines for Limited RAM

Your perspective belongs here.

1 question comment