RocksDB Column Families: Isolating Tenants Without Spinning Up New Processes

The Problem: One Database, Many Noisy Neighbors

When you put multiple customers into a single RocksDB instance, two things break quickly. First, their keys collide unless you manually prefix every key. Second, a tenant doing bulk imports can flush memtables and trigger compactions that slow down everyone else. The useful takeaway is that RocksDB column families give you isolated keyspaces inside the same process and storage engine, so you can separate tenants logically without running a separate database per customer.

What a Column Family Actually Isolates

A column family is a named keyspace within a RocksDB database. Each family owns its own memtable (the in-memory write buffer), its own set of SSTables on disk, and its own compaction configuration. That means a write-heavy tenant can be configured to flush more aggressively, while a read-heavy tenant can keep a larger write buffer and a different compaction style.

One nuance matters for multi-tenant planning: all column families in a single DB share the same write-ahead log (WAL). The WAL is the durability mechanism that records writes before they reach SSTables. Sharing it simplifies crash recovery, but it also means the log is a shared resource. If one tenant floods the WAL, recovery time can grow for everyone.

Worked Example: Two Tenants, One Database

The following C++ sketch creates two column families, writes a key into each, and scans only one family. Run it inside your application process; it requires read/write permission on the database directory. Replace <DB_PATH> with your target directory.

#include "rocksdb/db.h"
#include "rocksdb/options.h"
#include <iostream>
#include <string>
#include <vector>

int main() {
  const std::string db_path = "<DB_PATH>";

  rocksdb::Options options;
  options.create_if_missing = true;
  // Create column families that do not yet exist on disk.
  options.create_missing_column_families = true;

  std::vector<rocksdb::ColumnFamilyDescriptor> cf_descs;
  cf_descs.emplace_back("tenantA", rocksdb::ColumnFamilyOptions());
  cf_descs.emplace_back("tenantB", rocksdb::ColumnFamilyOptions());

  rocksdb::DB* db = nullptr;
  std::vector<rocksdb::ColumnFamilyHandle*> handles;
  rocksdb::Status s = rocksdb::DB::Open(options, db_path, cf_descs, &handles, &db);
  if (!s.ok()) {
    std::cerr << "Open failed: " << s.ToString() << std::endl;
    return 1;
  }

  rocksdb::WriteOptions wopt;
  s = db->Put(wopt, handles[0], "user:123", "Alice");
  s = db->Put(wopt, handles[1], "user:123", "Bob");

  // Scan only tenantA.
  rocksdb::ReadOptions ropt;
  rocksdb::Iterator* it = db->NewIterator(ropt, handles[0]);
  for (it->SeekToFirst(); it->Valid(); it->Next()) {
    std::cout << "tenantA key=" << it->key().ToString()
              << " value=" << it->value().ToString() << std::endl;
  }
  delete it;

  for (auto* h : handles) {
    db->DestroyColumnFamilyHandle(h);
  }
  delete db;
  return 0;
}

Expected check: the scan should print only tenantA key=user:123 value=Alice. If you see Bob, the iterator was created against the wrong handle. Risk: if you hard-code column family names, renaming them later requires a migration because RocksDB identifies families by name on open.

The Trade-off: Memory and Tuning Surface

The main cost is memory. Every column family allocates its own memtable and associated write-buffer resources. Ten families with 64 MB write buffers can consume 640 MB of RAM before anything is flushed, and that pressure can increase garbage-collection pauses in the host process. There is also a tuning cost: options such as write_buffer_size, max_write_buffer_number, and compaction_style are per-family, so a misconfiguration in one family can cause it to dominate I/O or memory while others starve.

A practical limitation: because families share the same WAL and the same DB directory, file-level backup scripts that copy only one family's SST files will miss data. Use rocksdb::Checkpoint or the BackupEngine to capture all families atomically.

How to Check Before You Commit

Before rolling this out, verify two things. First, confirm your RocksDB version supports the API you need; column families have been stable for many releases, but create_missing_column_families and Checkpoint should be present in any modern build (5.x and later). Second, run the prototype above and assert that a scan of tenantA returns only tenantA keys. Then monitor resident memory (/proc/<PID>/status on Linux or Activity Monitor on macOS) while writing to both families. If RSS stays within budget and the iterator check passes, column families are a safe isolation boundary for your workload.

To roll back the prototype, stop the process and remove <DB_PATH>; because the database is self-contained in that directory, deleting it returns the filesystem to its prior state. If the memory overhead is too high, fall back to a single column family with explicit key prefixes (tenantA:user:123) and rely on prefix bloom filters for isolation. You lose per-family compaction tuning, but you keep a single memtable and a simpler tuning surface.