Isolate RocksDB Workloads with Column Family Compaction and Memtable Tuning

Problem: One RocksDB Instance, Conflicting Workloads

When a single RocksDB instance stores both small, frequently updated metadata and large, append-heavy event data, compaction from the bulk workload can stall reads for the metadata workload. A practical fix is to use column families—key namespaces inside one database—so each workload gets its own memtable size, flush trigger, and compaction strategy. This keeps the operational simplicity of one DB while letting you tune each data shape independently.

How Column Families Isolate Behavior

A column family is a logical partition with its own memtable—the in-memory write buffer—SST (Sorted String Table) files, and compaction state. All families in an instance share the same Write-Ahead Log (WAL) and MANIFEST file, so a single crash recovery replays every family. Because each family has its own write buffer, a flush in one does not force a flush in another. Compaction also runs per family, so a universal compaction job on event data does not block a level compaction job on metadata.

Worked Configuration: Metadata vs. Event Data

The following C++ example opens one database with two families. The default family uses level compaction, a 64 MB write buffer, and a Bloom filter—a probabilistic structure that avoids disk reads for absent keys—to speed point lookups. The events family uses universal compaction, a 256 MB write buffer, and no Bloom filter to reduce memory overhead. Run this code on the application server that owns the data directory; the process needs write permission to the RocksDB path.

// Assumes a recent RocksDB C++ API; confirm exact signatures against your version.
// Headers: rocksdb/db.h, rocksdb/options.h
#include <vector>
#include <string>
#include "rocksdb/db.h"
#include "rocksdb/options.h"

// Placeholders: replace /var/lib/rocksdb_data with your actual DB path.
const std::string kDBPath = "/var/lib/rocksdb_data";

std::vector<rocksdb::ColumnFamilyDescriptor> column_families;

// Metadata CF: level compaction, 64 MB memtable, Bloom filter enabled.
rocksdb::ColumnFamilyOptions meta_options;
meta_options.compaction_style = rocksdb::kCompactionStyleLevel;
meta_options.write_buffer_size = 64 * 1024 * 1024;
meta_options.bloom_filter_bits_per_key = 10; // set to 0 to disable

column_families.emplace_back(
    rocksdb::kDefaultColumnFamilyName, meta_options);

// Events CF: universal compaction, 256 MB memtable, no Bloom filter.
rocksdb::ColumnFamilyOptions event_options;
event_options.compaction_style = rocksdb::kCompactionStyleUniversal;
event_options.write_buffer_size = 256 * 1024 * 1024;
event_options.bloom_filter_bits_per_key = 0; // portable way to disable

column_families.emplace_back("events", event_options);

rocksdb::DB* db = nullptr;
std::vector<rocksdb::ColumnFamilyHandle*> handles;

// Open the DB. handles[0] is default, handles[1] is "events".
rocksdb::Status s = rocksdb::DB::Open(
    rocksdb::Options(), kDBPath, column_families, &handles, &db);
if (!s.ok()) {
  // Handle open failure; do not proceed with writes.
  return;
}

// Write an event key into the events family only.
rocksdb::WriteOptions write_options;
s = db->Put(handles[1], write_options, "event_101", "payload_bytes");
// s.ok() should be true if the write succeeded.

// Read metadata from the default family using its handle.
std::string value;
s = db->Get(handles[0], rocksdb::ReadOptions(), "meta_key_1", &value);
// s.ok() is true only if the key exists in the default family.

// Clean up handles before closing the DB.
for (auto* handle : handles) {
  db->DestroyColumnFamilyHandle(handle);
}
delete db;

Key points in this snippet: ColumnFamilyOptions are supplied at open time through ColumnFamilyDescriptor. The bloom_filter_bits_per_key field is set to 0 to disable the filter, which is portable across RocksDB versions that may not expose a dedicated disable_bloom_filter() helper. Writes and reads are routed by the ColumnFamilyHandle* passed to Put and Get.

Limits and Common Mistakes

Logical Isolation, Not Physical Isolation

Column families do not provide a security boundary or a hard resource cap. They share the same disk, the same WAL, and the same background thread pool. A large write burst in the events family can still increase I/O latency for the default family because the WAL and compaction threads are shared. Treat column families as a tuning mechanism, not a multi-tenant sandbox.

Overhead of Too Many Families

Every column family adds at least one memtable, entries in the MANIFEST, and its own compaction state. There is no universal safe count; the right number depends on your write rate, memory budget, and disk. A common mistake is creating a family per user or per day without measuring the resulting MANIFEST and memory overhead. Monitor total memory and compaction queue depth as you add families, and remove unused ones.

Changing Options After Creation

Options such as compaction style and write buffer size are part of the ColumnFamilyOptions recorded in the MANIFEST. You can adjust some parameters at runtime with AlterColumnFamilyOptions, but structural changes—such as switching from level to universal compaction—may require creating a new column family and rewriting data. Plan the initial compaction style carefully.

Dropping a Family Is Asynchronous

DropColumnFamily removes the family from the MANIFEST and returns quickly, but obsolete SST files are deleted in the background. Disk usage can remain high until cleanup finishes. If disk space is tight, check available space before dropping large families.

Atomic Batches, Not Serializable Isolation

A WriteBatch can include operations for multiple column families and will apply atomically: either all writes in the batch are visible or none are. This is not the same as serializable isolation across separate batches. If two concurrent writers modify the same key space in different families, RocksDB still resolves write-write conflicts at the key level, but cross-family transactional semantics are not provided by the column family mechanism alone.

Verification and Diagnostics

After configuring the families, verify that isolation behaves as expected.

1. Handle-based isolation check: Write a key to handles[1], then call Get with handles[0]. The status should be rocksdb::Status::NotFound(), confirming the key is not visible in the wrong namespace.
2. Family inventory: Call the static method rocksdb::DB::ListColumnFamilies(rocksdb::DBOptions(), kDBPath, &family_names) and confirm that default and events are present.
3. Per-family size and files: Retrieve ColumnFamilyMetaData for each handle and inspect the estimated size and file count. This confirms that writes are landing in the intended family and that each tracks its own storage.
4. Compaction independence: Use db->GetIntPropertyCF(handle, "rocksdb.num-running-compactions") for each handle. Under mixed load, the values should change independently, showing that each family schedules its own compaction jobs.

If the wrong-handle Get returns a value instead of NotFound, you are likely passing the wrong ColumnFamilyHandle* or mixing default and named family handles. If both families report the same running compaction count at all times, confirm that the write buffer sizes and compaction priorities are actually different and that background threads are not saturated by a single family.