RocksDB Block Cache memory growth with shared Column Families

Answer

The shared Block Cache does enforce the configured capacity for the data it tracks, and moving to per-Column Family caches changes isolation at the cost of overhead. The RSS vs rocks.block_cache_usage gap you see is expected and is not a failure of eviction.

Confirmed facts

• A shared BlockBasedTableOptions.block_cache is a single cache instance referenced by multiple Column Families. Insert charge is the block size plus internal overhead, and eviction is global LRU across all CFs.
• rocks.block_cache_usage reports the sum of charges held in the cache. It does not include MemTable memory, WAL, or process allocator overhead.
• RocksDB uses a custom allocator by default. Even after a block is evicted from the cache, the allocator typically retains the memory for reuse and does not return it to the OS immediately. RSS therefore lags behind cache usage.
• Index and filter blocks are cached in the same Block Cache. Metadata structures for the cache itself, per-entry handles, and allocator fragmentation contribute to RSS beyond the reported usage.

Likely explanation for your symptoms

Under high concurrency and heavy random reads across diverse Column Families, simultaneous insertions cause rapid LRU churn. The cache enforces the size limit on a per-insert basis, so eviction does occur, but:

• Memory is not returned to the OS on eviction; it stays in the allocator arena. RSS stays high while rocks.block_cache_usage oscillates near the cap.
• LRU contention and varied block sizes across CFs increase internal fragmentation. The allocator cannot reuse freed pages efficiently, so resident pages accumulate.
• Pinned usage from ongoing reads, rocks.block_cache_pinned_usage, temporarily holds blocks outside normal LRU accounting.

Shared vs isolated caches

Isolated caches per Column Family prevent one CF from evicting another CF's hot data and make per-CF limits explicit. The trade-off is higher total overhead because each cache maintains its own LRU lists, hash tables and allocator arenas. Total RSS often rises with isolation, even though per-CF predictability improves.

Steps needed for this case

1. Baseline both metrics together:

   cat /proc/self/status | grep -E 'VmRSS|VmHWM'

   Compare with rocks.block_cache_usage and rocks.block_cache_pinned_usage from RocksDB statistics.
2. Check allocator behavior. If you are using the default allocator, expect RSS to stay elevated after eviction. Test with options.allow_mmap_reads and allocator settings only after a baseline is captured.
3. Inspect cache pressure signals: rocks.block_cache_hit_rate, rocks.block_cache_index_usage, and eviction counts per CF. High pinned usage indicates reads are holding memory.
4. If isolation is required, create separate caches with explicit capacities that sum to less than the current shared cap to account for duplicated overhead. Measure RSS before and after the change under the same workload.

One diagnostic detail that changes the recommendation: which allocator RocksDB is built with and whether Cache::NewLRUCache is configured with strict capacity and no high-priority partition. The allocator determines how quickly RSS can fall after eviction, and high-priority partitions can make the cache appear to exceed the nominal limit.