RocksDB Column Families: One Database, Independently Tuned Data Sets

The problem: one workload, two very different access patterns

Say you're storing user sessions and audit events in the same service. Sessions are small, hot, and updated constantly. Audit events are append-only, written in bursts, and read rarely. If both live in a single RocksDB key space, they share one compaction policy, one block cache budget, and one write path — so the compaction strategy that's right for sessions actively hurts the audit log, and vice versa.

RocksDB's answer is column families: logically separate key spaces inside a single database instance, each with its own options. The useful takeaway: you get per-data-set tuning (compaction style, compression, memtable sizing) without running multiple database processes or splitting your WAL and backup story across directories.

What a column family actually is

Every RocksDB database has a column family named default, whether you ask for it or not. Additional families are created at open time or at runtime with CreateColumnFamily. Each family gets:

• Its own memtables and SST files, so writes to one family never trigger compaction in another.
• Its own ColumnFamilyOptions — compaction style, compression, block table factory, bloom filters.
• Its own statistics, visible per family in rocksdb.stats.

All families share the database's write-ahead log and the overall DBOptions (things like the WAL directory and total background thread pools). That sharing is the point: atomic multi-family writes via WriteBatch are possible, and operational concerns like backups stay unified.

One detail that bites people: you interact with families through ColumnFamilyHandle objects, and those handles own native resources. Every handle must be destroyed before the DB object is closed — leaking one is a reliable way to crash at shutdown.

Worked example: two families, two compaction styles

This C++ sketch (against RocksDB 7.x/8.x headers, built with -lrocksdb) opens a database with a second family tuned differently from the default. Run it as any user with write access to the data directory; no special privileges needed.

#include <rocksdb/db.h>
#include <rocksdb/options.h>
#include <vector>
#include <string>
#include <cassert>

using namespace ROCKSDB_NAMESPACE;

int main() {
  const std::string path = "/tmp/cf_demo";

  // 1. Describe the families we expect to exist.
  ColumnFamilyOptions session_opts;
  session_opts.compaction_style = kCompactionStyleLevel;
  session_opts.compression = kLZ4Compression;  // hot data: cheap compression

  ColumnFamilyOptions audit_opts;
  audit_opts.compaction_style = kCompactionStyleUniversal;
  audit_opts.compression = kZSTD;              // cold data: max ratio

  std::vector<ColumnFamilyDescriptor> families = {
    {kDefaultColumnFamilyName, session_opts},
    {"audit", audit_opts},
  };

  // First run: create the "audit" family explicitly.
  {
    DB* raw;
    Options o; o.create_if_missing = true;
    Status s = DB::Open(o, path, &raw);
    assert(s.ok());
    std::unique_ptr<DB> db(raw);
    ColumnFamilyHandle* h = nullptr;
    s = db->CreateColumnFamily(audit_opts, "audit", &h);
    if (!s.ok() && !s.IsInvalidArgument()) assert(false); // exists = fine
    delete h;  // handle must go before db closes
  }

  // 2. Open with both families and write to each.
  std::vector<ColumnFamilyHandle*> handles;
  DBOptions db_opts;
  DB* raw;
  Status s = DB::Open(db_opts, path, families, &handles, &raw);
  assert(s.ok());
  std::unique_ptr<DB> db(raw);

  WriteOptions wo;
  db->Put(wo, handles[0], "session:42", "{...}");   // default family
  db->Put(wo, handles[1], "audit:000001", "login"); // audit family

  std::string v;
  s = db->Get(ReadOptions(), handles[1], "audit:000001", &v);
  assert(s.ok() && v == "login");

  // 3. Cleanup order matters: handles first, then the DB.
  for (auto* h : handles) delete h;
  db.reset();
  return 0;
}

The pattern to copy is the two-phase open: create the family once (guarding against "already exists"), then reopen listing every family. RocksDB refuses to open a database if you don't list all existing families, unless you deliberately open in a mode that skips unknown ones — a common first-run surprise.

Verifying the isolation actually happened

Don't take the option objects on faith. Two practical checks:

1. Per-family stats. After writing enough data to force flushes, call db->GetProperty(handles[1], "rocksdb.stats", &out) and confirm the audit family shows its own compaction section with universal-style output, separate from the default family. If both sections look identical, your options didn't land on the family you think they did.
2. Filesystem layout. Each family's SST files carry distinct file numbers, but the simplest signal is the MANIFEST and the live-file set: after DropColumnFamily(handle) plus handle destruction, the family's files are scheduled for removal and disk usage in the data directory should drop once obsolete-file deletion runs. If it doesn't, a leaked handle is the usual suspect.

For a quicker sanity check without writing code, rocksdb_bench supports exercising column families and reports per-family compaction stats, which is handy for validating a compaction-style choice before committing to it.

Trade-offs worth knowing before you reach for families

• Metadata and file-handle overhead scales with family count. A handful of families with genuinely different workloads is the sweet spot. Dozens of families created "just to organize keys" adds memory per memtable and more open files, and can degrade read performance. If the data doesn't need different options, a key prefix is cheaper.
• Runtime creation isn't free. CreateColumnFamily during traffic is supported, but doing it in a hot loop (e.g., one family per tenant) burns file descriptors and MANIFEST churn. Create families at startup where possible.
• Drops need cleanup discipline. DropColumnFamily removes the family from the manifest, but stale files linger if handles leak or the process dies mid-operation — check disk usage after drops, not just the API return code.
• Mis-tuned per-family compaction shows up as disk spikes. Universal compaction on a family you expected to be small can double space usage during compaction. Monitor rocksdb.stats per family after any option change.

Closing

Column families turn "which compaction policy do we pick?" from a database-wide compromise into a per-data-set decision, while keeping one WAL and one operational footprint. Start by listing your data sets and their write/read shapes; if two of them genuinely want different compaction or compression, that's your signal to split them into families — then verify with per-family stats before and after load.