Bounding RocksDB Memory Across Column Families with a Shared WriteBufferManager

If you run RocksDB with several column families in one process, the default memory model can surprise you. Each column family gets its own write buffer, and nothing in the per-CF settings bounds the total. Ten column families with a 64 MB write buffer can allocate 640 MB of memtables before any global check kicks in. In a container with a 1 GB limit, that is an OOM kill waiting for the next burst of writes.

The useful takeaway: assign a single shared WriteBufferManager to the column families and DB instances that live in the same process, and size its budget below the container limit. That turns unbounded per-CF allocation into a process-wide budget that triggers flushes or stalls instead of an OOM.

Why per-CF write buffers multiply

RocksDB writes incoming data first into an in-memory structure called a memtable. When the memtable fills, it becomes immutable and a background flush writes it to an SST file. The knob write_buffer_size controls how large one memtable can grow, and max_write_buffer_number controls how many memtables a column family can hold before writes stall or fail. These are per-column-family settings.

Nothing in the default configuration aggregates those numbers. If you open five column families with the same options, the process can hold five times the memtable memory. The same applies when you open multiple DB instances in one process: each DB has its own column families, and each column family has its own buffers. Internal estimates of memory usage can also drift from actual RSS when large values, blobs, or pinned iterators are involved, so the gap between predicted and observed memory can widen under load.

How WriteBufferManager coordinates memory

A WriteBufferManager is a shared coordinator for memtable memory. You create one instance with a total budget and attach it to the options of the column families or DBs that should share that budget. When the aggregate memtable usage across all attached column families exceeds the budget, RocksDB can flush memtables and, depending on the manager's configuration, stall incoming writes until memory is released.

The exact constructor flags and defaults—especially around whether stalls are allowed and when flushing is triggered—have changed across RocksDB releases. Treat the manager's behavior as version-sensitive and confirm the semantics in the headers shipped with your build before relying on it in production.

A practical configuration pattern

Start by creating one manager with a budget comfortably below your container limit, then share it across column families while keeping per-CF buffers small enough that a single CF cannot consume the entire budget on its own.

// Verify the exact constructor signature and flag names in your installed
// include/rocksdb/write_buffer_manager.h before compiling.
std::shared_ptr<rocksdb::WriteBufferManager> shared_wb =
    rocksdb::WriteBufferManager(
        /*total budget=*/512 * 1024 * 1024,
        /*block cache=*/nullptr,
        /*allow stalls=*/true);

rocksdb::Options opts;
opts.write_buffer_manager = shared_wb;
opts.write_buffer_size = 64 * 1024 * 1024;
opts.max_write_buffer_number = 2;

// Apply opts to every column family that should draw from the same budget.

To confirm the available flags in your installed version, inspect the header on the build machine or in your container image:

grep -n "WriteBufferManager" \
  /usr/local/include/rocksdb/write_buffer_manager.h

Run the process under a fixed memory ceiling so the test reflects production constraints:

docker run --rm --memory=2g --memory-swap=2g \
  your-image ./mixed_write_bench

Expected checks: the process should stay below the container limit, the flush counter should rise as the budget is hit, and the write-stall counter should increase only if you allow stalls. If RSS still climbs past the budget, the manager is not the only consumer—see the limitations below.

Trade-offs and blind spots

A smaller shared budget reduces OOM risk but increases flush frequency, write amplification, and the chance of write stalls. A larger budget improves throughput but can reintroduce memory pressure. There is no universal percentage; the right budget depends on your write rate, compaction speed, and how much latency your application can tolerate.

More importantly, WriteBufferManager does not cap everything. It does not bound block cache, OS page cache, compaction scratch memory, or all index and filter memory. Treat it as one layer in a stack: size the block cache separately, tune compaction concurrency, and set the container limit to a value that leaves headroom for the parts the manager does not see.

Verifying the result

Do not trust internal accounting alone. Run a sustained mixed write workload with and without the shared manager under the same container limit. Record external RSS, the memtable-flush counter, the write-stall counter, and block-cache usage from RocksDB statistics or column-family properties. Compare the two runs: the managed run should show more flushes and lower peak RSS. If stalls appear, measure the latency impact before deciding whether to raise the budget or reduce per-CF buffer sizes.

RocksDB statistics and properties give you a fast signal, but confirm with process RSS from outside the process—ps, cgroup memory counters, or your container metrics—because internal estimates can diverge from actual usage.

Actionable next step

If you run multiple column families or DB instances in one process, create one shared WriteBufferManager sized below your container limit, attach it to every column family that should share the budget, and set per-CF write_buffer_size small enough that no single column family can exhaust the budget alone. Then verify with a fixed-memory test run and adjust the budget based on observed flush and stall rates.