Question
Write stall activation criteria for mutable memtable memory limits
Zo Byte
0 reputation · 20 Apr 2022, 08:46 UTC
60.1K views
Loading…
0 reputation · 20 Apr 2022, 08:46 UTC
When configuring RocksDB, the write stall mechanism is designed to pause writes when memory pressure from mutable memtables exceeds configured limits. The documentation states that a stall occurs if the total memory used by all mutable memtables surpasses write_buffer_limit or if the number of immutable memtables awaiting flush reaches max_write_buffer_number. However, it is unclear whether the stall evaluation considers the aggregate size of mutable memtables only, or also checks each individual memtable against write_buffer_size. Additionally, the interaction between max_write_buffer_number and min_write_buffer_number_to_merge in triggering stalls is not explicitly detailed.
Understanding the precise condition that initiates a stall is important for tuning write throughput without risking out‑of‑memory situations.
Does the stall trigger when the cumulative size of all mutable memtables exceeds write_buffer_limit, regardless of the size of any single memtable?
Does max_write_buffer_number influence the stall threshold independently of total memtable memory?
Is the stall condition evaluated per memtable or only on the aggregate mutable memtable usage?
34775 reputation · 20 Apr 2022, 19:33 UTC
In RocksDB, write stalls are triggered by two primary independent conditions: aggregate memory usage and the count of pending flushes. The stall mechanism does not evaluate individual memtable sizes against the global limit; it monitors the state of the entire memtable pool.
A write stall occurs when the cumulative size of all mutable memtables exceeds the write_buffer_limit. This is an aggregate check. While an individual memtable's transition from mutable to immutable is triggered by write_buffer_size, the stall itself is only activated if the total memory used across all mutable memtables surpasses the global limit.
The max_write_buffer_number acts as a separate, independent trigger. If the number of immutable memtables awaiting a flush to disk reaches this threshold, RocksDB will initiate a stall regardless of whether the total memory usage is below the write_buffer_limit. This prevents the system from accumulating too many pending flushes, which could lead to an Out-of-Memory (OOM) event during a crash recovery or a massive spike in memory usage.
The interaction between these settings can be summarized as follows:
rocksdbwrite_buffer_size determines when a single memtable is full and becomes immutable.write_buffer_limit monitors the sum of all mutable memtables to trigger a stall.max_write_buffer_number monitors the queue depth of immutable memtables to trigger a stall.Regarding min_write_buffer_number_to_merge, this setting typically influences the flush decision (whether to merge multiple immutable memtables into one SST file) rather than acting as a direct trigger for the write stall mechanism itself.
To verify these behaviors in your specific environment (assuming RocksDB 6.x or newer), you can use the following scoped tests:
write_buffer_limit and a high max_write_buffer_number. Perform high-volume writes and monitor the rocksdb.memtable.stall.micros statistic to confirm stalls trigger based on total memory.write_buffer_limit but a very low max_write_buffer_number (e.g., 2). Observe if writes stall even when total memory usage is well below the limit, indicating the count-based trigger is active.Diagnostic Detail Needed: Are you using a Column Family configuration? If so, the write_buffer_limit is shared across all column families, which can lead to more frequent stalls if one family consumes the majority of the shared pool.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 20 Apr 2022, 14:16 UTC
The existing answer correctly identifies the two independent stall triggers. One practical nuance: min_write_buffer_number_to_merge does not directly cause stalls, but it controls how many mutable memtables are merged before a new immutable memtable is created. A lower value (e.g., 1) causes more frequent immutable memtable creation, which can make the max_write_buffer_number limit hit sooner under sustained write pressure. Conversely, a higher value batches more data per immutable memtable, potentially delaying the count-based stall but increasing per-flush I/O size. This trade-off is worth considering when tuning for write throughput versus stall frequency.