Isolate Workloads in RocksDB Using Column Families with Custom Memtable and Compaction Settings

Why use column families for workload isolation

RocksDB column families let you keep logically separate key‑value spaces inside a single database instance. Each family can have its own memtable size, write buffer count, and compaction style while sharing the same underlying storage and write‑ahead log (WAL). This makes it possible to tune a read‑heavy workload (e.g., user profiles) differently from a write‑heavy workload (e.g., application logs) without running two separate DB instances.

Configuration example

The following C++ snippet shows how to open a RocksDB instance with two column families: a "default" family for general data and a "logs" family for high‑volume append‑only logs. Each family gets its own write buffer size and compaction style.

#include 
#include 
#include 
#include 
#include 
#include 

int main() {
    rocksdb::DB* db = nullptr;
    rocksdb::Options db_options;
    db_options.create_if_missing = true;
    db_options.IncreaseParallelism();
    db_options.OptimizeLevelStyleCompaction();  // default compaction style

    // Options for the default column family
    rocksdb::ColumnFamilyOptions cf_default_options;
    cf_default_options.write_buffer_size = 64 << 20;   // 64 MB
    cf_default_options.max_write_buffer_number = 2;   // up to 128 MB memtable total
    cf_default_options.compaction_style = rocksdb::kLevelCompaction;

    // Options for the logs column family
    rocksdb::ColumnFamilyOptions cf_logs_options;
    cf_logs_options.write_buffer_size = 16 << 20;    // 16 MB
    cf_logs_options.max_write_buffer_number = 2;    // up to 32 MB memtable total
    cf_logs_options.compaction_style = rocksdb::kUniversalCompaction;  // better for write‑heavy

    std::vector column_families;
    column_families.emplace_back(rocksdb::ColumnFamilyDescriptor(
        rocksdb::kDefaultColumnFamilyName, cf_default_options));
    column_families.emplace_back(rocksdb::ColumnFamilyDescriptor(
        "logs", cf_logs_options));

    std::vector handles;
    rocksdb::Status status = rocksdb::DB::Open(
        db_options, "/tmp/rocksdb_example", column_families, &handles, &db);
    if (!status.ok()) {
        std::cerr << "Open failed: " << status.ToString() << std::endl;
        return 1;
    }

    // Example writes using the obtained handles
    rocksdb::WriteOptions write_opt;
    db->Put(write_opt, handles[0], "user:1000", "Alice");   // default CF
    db->Put(write_opt, handles[1], "log:2026-09-17", "event"); // logs CF

    // Remember to close handles before closing the DB
    for (auto* h : handles) delete h;
    delete db;
    return 0;
}

Verifying the configuration

After the DB is running you can confirm that each family respects its memtable settings and that compactions are logged independently.

• Memtable usage: Call db->GetProperty("size-all-mem-tables"). The returned value should be close to the sum of (write_buffer_size × max_write_buffer_number) for both families (≈ 192 MB in the example).
• Compaction logs: Enable the RocksDB info log (db_options.info_log_level = rocksdb::InfoLogLevel::INFO_LEVEL) and look for lines containing "ColumnFamily" and "compaction". You should see separate entries for the default and logs families, showing that each follows its configured compaction style.

Limits and common mistakes

While column families provide isolation, they share certain resources, which can lead to unexpected behavior if not managed correctly.

• Shared WAL: All families write to the same write‑ahead log. A burst of writes in one family can increase WAL sync latency and affect the other families. Monitor the log sync time (wal_sync_micros property) to detect contention.
• Handle leaks: Every column family obtained from DB::Open must be deleted before calling DB::Close. Forgetting to do so leaks memory and can prevent clean shutdown.
• Snapshot misuse: A snapshot taken with one column family handle does not guarantee a consistent view across families. For cross‑family reads obtain a single snapshot via db->GetSnapshot() and use it with all relevant handles.
• Too many families: Each family adds metadata overhead (manifest entries, lock objects). Keeping the number of families below ~10 is a practical rule‑of‑thumb to avoid noticeable performance impact.

Practical checklist

1. Define separate ColumnFamilyOptions for each workload, setting write_buffer_size and max_write_buffer_number to match the desired memtable budget.
2. Choose a compaction style that fits the access pattern (e.g., kLevelCompaction for read‑heavy, kUniversalCompaction or kFIFOCompaction for write‑heavy).
3. Open the DB with a vector of ColumnFamilyDescriptor objects and store the returned handles.
4. After loading data, verify memtable usage with the "size-all-mem-tables" property and check the info log for family‑specific compaction entries.
5. Monitor WAL sync latency (wal_sync_micros) to ensure one family’s write burst is not degrading the others.
6. Always delete column family handles before closing the DB, and use a single snapshot handle when reading across families.