Which memory limit actually isolates RocksDB column families sharing a block cache?

Answer

With a shared block cache there is no hard per-family memory limit that isolates cache residency. A WriteBufferManager limits memtable / write buffer memory, not block cache usage. True per-family isolation of cached data requires separate Cache instances per column family.

Confirmed facts for RocksDB 7.x-8.x

• WriteBufferManager = memtable isolation, not cache isolation. It caps total write buffer memory across column families and provides backpressure for writes. It does not cap how much of a shared block cache a single family can occupy and does not prevent one family from evicting another's blocks.
• Shared block cache = no per-family quota. When DBOptions.block_cache is shared, eviction is LRU across all families. A hot family can displace blocks belonging to cold families. No per-family budget is enforced.
• Separate Cache instances = hard isolation. Creating a distinct Cache per ColumnFamilyOptions gives each family its own memory budget. This is the only mechanism that provides observable hard isolation for cached data.

Likely explanation vs confirmed behavior

Per-family block cache priority and cache_index_and_filter_blocks are soft hints, not budgets.

• BlockBasedTableOptions.block_cache_priority is a soft eviction preference within the shared Cache. It can bias which entries are evicted first, but it does not enforce a memory cap.
• cache_index_and_filter_blocks controls what metadata is placed in the shared cache. Changing it changes churn and residency, not isolation.

Memtable pressure and block cache churn are independent layers.

Distinguish memtable pressure from block cache churn

Memtable pressure:

• WriteBufferManager stats: total memory used, stall conditions
• Per-CF write_buffer_size and num immutable memtables
• Stalls: write stall, flush stall metrics

Block cache churn:

• Cache hit rate, miss rate
• Evictions, bytes read, bytes written to cache
• Per-CF block cache usage is not tracked with a shared cache; only aggregate cache stats are available

Verification steps for this case

1. Inspect configuration: DBOptions for WriteBufferManager size, ColumnFamilyOptions write_buffer_size, and whether block_cache is set in DBOptions vs per ColumnFamilyOptions.
2. Check runtime stats for memtable: memtable size, stall conditions, WriteBufferManager utilization.
3. Check runtime stats for block cache: hit rate, evictions, bytes read.
4. Run a hot read workload on one family while monitoring cache residency and hit rate of a cold family to observe eviction interaction.

One missing diagnostic detail that changes the recommendation: the exact RocksDB release in use and whether the block cache is currently shared via DBOptions or already assigned per column family. Option names and Cache API behavior are version sensitive in 7.x-8.x.