Using RocksDB Column Families to Isolate Workloads in a Single DB Instance

Desired outcome

Create a RocksDB instance that holds multiple, independent key‑spaces (column families) so that writes to one space do not interfere with the compaction, memtable size, or write‑ahead log (WAL) buffering of another. This lets you tune each workload separately while still sharing the same underlying storage and benefiting from a single snapshot/restore point.

Prerequisites

• RocksDB library built and linked against your application (version 6.x or later recommended).
• Development environment with C++11 support and a compiler that can handle RocksDB headers.
• Write permission on the directory where the database will be stored ().
• Basic familiarity with RocksDB Options and ColumnFamilyOptions concepts.

Procedure

1. Include headers and set up the DB options

   #include <rocksdb/db.h>
   #include <rocksdb/options.h>
   #include <rocksdb/cf.h>
   
   using namespace rocksdb;
   
   Options db_options;
   db_options.create_if_missing = true;
   // Optional: tune global settings that apply to all families
   db_options.IncreaseParallelism();
   db_options.OptimizeLevelStyleCompaction();
   

2. Define ColumnFamilyOptions for each workload

   Here we create two families: one for high‑throughput writes (cf_writes) and another for read‑heavy metadata (cf_meta). Adjust the settings to match your workload.

   ColumnFamilyOptions write_cf_options;
   write_cf_options.optimize_for_point_lookup = true;   // good for short lookups
   write_cf_options.write_buffer_size = 64 << 20;      // 64 MiB memtable
   write_cf_options.target_file_size_base = 128 << 20; // 128 MiB SST files
   
   ColumnFamilyOptions meta_cf_options;
   meta_cf_options.optimize_for_point_lookup = false;  // larger values, range scans
   meta_cf_options.write_buffer_size = 128 << 20;     // 128 MiB memtable
   meta_cf_options.target_file_size_base = 256 << 20; // 256 MiB SST files
   

3. Open the DB and create the column families

   The first family opened is the "default" family; additional families are created via CreateColumnFamily and must be kept open with a ColumnFamilyHandle*.

   std::vector<ColumnFamilyDescriptor> column_families;
   column_families.emplace_back(ColumnFamilyDescriptor(
       kDefaultColumnFamilyName, ColumnFamilyOptions()));
   column_families.emplace_back(ColumnFamilyDescriptor(
       "cf_writes", write_cf_options));
   column_families.emplace_back(ColumnFamilyDescriptor(
       "cf_meta", meta_cf_options));
   
   std::vector<ColumnFamilyHandle*> handles;
   Status s = DB::Open(db_options, "", column_families, &handles, &db);
   if (!s.ok()) {
       fprintf(stderr, "Failed to open DB: %s\n", s.ToString().c_str());
       return;
   }
   
   // Assign handles for easier use later
   ColumnFamilyHandle* default_handle = handles[0];
   ColumnFamilyHandle* writes_handle  = handles[1];
   ColumnFamilyHandle* meta_handle    = handles[2];
   

4. Write and read using specific handles

   All RocksDB APIs that accept a ColumnFamilyHandle* route the operation to that family’s memtable, WAL buffer, and compaction thread.

   // Write to the write‑optimized family
   s = db->Put(WriteOptions(), writes_handle, "user:123", "activity_log");
   assert(s.ok());
   
   // Write to the meta‑optimized family
   s = db->Put(WriteOptions(), meta_handle, "user:123:profile", "{name:Alice}");
   assert(s.ok());
   
   // Read from each family
   std::string val;
   s = db->Get(ReadOptions(), writes_handle, "user:123", &val);
   if (s.ok()) printf("Writes CF value: %s\n", val.c_str());
   
   s = db->Get(ReadOptions(), meta_handle, "user:123:profile", &val);
   if (s.ok()) printf("Meta CF value: %s\n", val.c_str());
   

5. Optional: Drop a column family when it is no longer needed

   Dropping removes the family’s metadata but leaves the underlying files intact until a compaction purges them. You must close all handles to the family before dropping.

   // Close the handle (important before dropping)
   db->DestroyColumnFamilyHandle(meta_handle);
   meta_handle = nullptr;
   
   // Drop the family from the DB
   s = db->DropColumnFamily(meta_handle); // handle can be null after destroy
   assert(s.ok());
   

Expected checks

• Separate memtables: After writing a known number of keys to each family, query the property rocksdb.cur-size-all-mem-tables per family. The sum should roughly equal the size you expect for each family’s write buffer.
• Independent compaction stats: Use DB::GetProperty with rocksdb.num-files-at-level0 (or rocksdb.num-files-at-level1) for each handle. Different families should show divergent counts if their write rates differ.
• Snapshot isolation: Take a snapshot via db->GetSnapshot() after writing to family A, then write new keys to family B. Reads from the snapshot using family A’s handle must not see the new B‑family keys.
• Crash recovery: Kill the process, restart, reopen the DB with the same options and handles, and iterate each family to confirm that all previously written key‑value pairs are present.

Recovery options

• If the WAL becomes corrupted, RocksDB will refuse to start; you must restore from a backup or let the DB start with WAL_ttl_seconds=0 and WAL_size_limit_MB=0 to disable WAL reliance (risking recent writes).
• If a specific column family’s SST files are corrupted (detected via VerifyChecksum in the manifest), you can drop that family (as shown above) and recreate it with the same name. Other families remain unaffected because they store separate file sets.
• To mitigate excessive memory overhead from many families, monitor rocksdb.cur-size-all-mem-tables and consider merging low‑traffic families or increasing the max_background_compactions to keep file levels tidy.