Choosing a RocksDB Compaction Style for Write-Heavy Embedded Workloads

The decision you actually face

You have an embedded RocksDB instance ingesting writes faster than it can comfortably compact them: the write stall counters creep up, L0 files pile up, and the default leveled compaction is burning I/O rewriting data that will be rewritten again tomorrow. The question is whether to stay with leveled compaction and tune it, switch to universal compaction to cut write amplification, or — if your data has a natural lifetime — move to FIFO and let old files drop on their own.

The takeaway up front: for a write-heavy workload with bounded data size, universal compaction usually wins on sustained write throughput; leveled stays the right choice when reads matter or disk is tight; FIFO is only correct when losing old data is a feature, not a bug. The rest of this guide walks through the constraints and a concrete way to validate the choice on your own hardware.

Constraints to pin down first

Before comparing options, answer four questions, because each compaction style optimizes a different corner of the space/write/read amplification triangle:

• Write-to-read ratio. If reads are rare or served from cache, you can afford more overlapping sorted runs (universal). If point lookups dominate, leveled's single-run-per-level layout wins.
• Disk headroom. Leveled targets roughly 10% space overhead. Universal can temporarily spike well above that during large merges unless you bound it. FIFO is capped by design.
• Data lifetime. Time-series, logs, and cache-like data with an expiry policy are FIFO candidates. Anything you must keep is not.
• Write budget. Leveled compaction commonly produces write amplification in the 10–30x range. Universal trades some of that for space. If your storage is IOPS-limited (cloud volumes, QLC SSDs), this matters more than raw throughput.

Comparing the three supported styles

Property	Leveled (default)	Universal	FIFO
Write amplification	Highest (~10–30x typical)	Lowest of general-purpose	Near zero (deletion)
Space amplification	Lowest (~10% target)	Higher; bound via options	Fixed cap via size limit
Read amplification	Lowest (one run per level)	Higher (overlapping runs)	Moderate; bloom filters help
Data loss risk	None	None	Oldest files deleted
Best fit	Mixed or read-heavy	Write-heavy, size-bounded	Time-series, caches

Trade-offs in practice

Leveled organizes data into levels L0 through LN, with each level roughly max_bytes_for_level_multiplier times larger than the previous. Compaction repeatedly merges one level into the next, which is why writes get rewritten many times. It gives the most predictable read latency and the smallest footprint, which is why it is the default.

Universal keeps a set of sorted runs and merges them when their relative sizes cross compaction_options_universal.size_ratio (percent) or when space amplification exceeds max_size_amplification_percent. Fewer, larger merges mean less rewriting, so sustained write throughput improves — but a big merge can temporarily nearly double disk usage, and reads may need to check several runs. Always set max_size_amplification_percent explicitly; without a bound, a write burst can fill the disk.

FIFO barely compacts at all. Once total size exceeds compaction_options_fifo.max_table_files_size (or files age past ttl), the oldest SST files are simply deleted. This is the cheapest possible write path, but deletion is silent and permanent. If you cannot answer "yes" to "is it fine if the oldest data disappears?", FIFO is off the table.

Concrete configuration: universal for a write-heavy store

The following C++ snippet sets up universal compaction with an explicit space-amplification bound. Run it in your application where you construct rocksdb::Options; it requires the same privileges as opening the database. Values are deliberate — do not rely on defaults, since they shift between releases.

rocksdb::Options options;
options.create_if_missing = true;
options.compaction_style = rocksdb::kCompactionStyleUniversal;

// Bound temporary space blowup during large merges.
options.compaction_options_universal.max_size_amplification_percent = 200;
options.compaction_options_universal.size_ratio = 10;

// Keep L0 from stalling writes too early under bursty ingest.
options.level0_file_num_compaction_trigger = 4;
options.level0_slowdown_writes_trigger = 20;
options.level0_stop_writes_trigger = 36;

// Collect stats for validation.
options.statistics = rocksdb::CreateDBStatistics();

rocksdb::DB* db;
rocksdb::Status s = rocksdb::DB::Open(options, "/var/lib/myapp/db", &db);
if (!s.ok()) { /* handle error */ }

For comparison, the equivalent leveled baseline is:

options.compaction_style = rocksdb::kCompactionStyleLevel;
options.level0_file_num_compaction_trigger = 4;
options.max_bytes_for_level_base = 256 * 1024 * 1024;  // 256 MB L1
options.max_bytes_for_level_multiplier = 10;

Two cautions before you deploy either. First, changing compaction style on an existing database is not a free operation — the LSM layout differs, so plan for a full compaction after the switch (or migrate to a fresh DB) and verify with a consistency check. Second, confirm the exact option names and defaults in include/rocksdb/options.h of the version you actually ship; this guide assumes a recent 7.x/8.x-era API.

Validating the choice with db_bench

Do not pick a style from a table alone. RocksDB ships db_bench (build it from the source tree with make db_bench, or use the packaged binary). Run it on the same storage class as production, as the same user the service will run as:

# Universal candidate
./db_bench --db=/tmp/bench_univ --benchmarks=fillrandom,readwhilewriting \\
  --num=50000000 --compaction_style=2 \\
  --statistics=1 --stats_interval_seconds=30

# Leveled baseline
./db_bench --db=/tmp/bench_lev --benchmarks=fillrandom,readwhilewriting \\
  --num=50000000 --compaction_style=0 \\
  --statistics=1 --stats_interval_seconds=30

(--compaction_style takes the enum ordinal: 0 = leveled, 1 = universal, 2 = FIFO — verify against your version's db_bench help output.) Adjust --num so the dataset exceeds memory; a benchmark that fits in the block cache tells you nothing about compaction.

Compare three things between runs:

1. Write amplification from the statistics output: ratio of compaction bytes written to user bytes written (rocksdb.compaction.times.micros and the bytes-written counters).
2. Read p99 latency during readwhilewriting — this is where universal's extra sorted runs show their cost.
3. Peak on-disk size via du on the DB directory sampled during the run, confirming universal stays within your max_size_amplification_percent budget.

Finally, inspect the LOG file's compaction summary lines (per-level file counts and sizes) to confirm the LSM shape matches what you configured — a universal DB should show a handful of large sorted runs, not a tidy level pyramid. If the numbers do not clearly favor universal, stay with leveled: it is the best-tested path and the cheapest to operate.

Limitations

Amplification figures quoted here are widely cited ranges, not guarantees; your key sizes, value sizes, and overwrite patterns shift them substantially. FIFO's TTL and size enforcement operate at file granularity, so data can outlive its nominal TTL until its file becomes the oldest. And any benchmark on empty disks overstates steady-state performance — rerun candidates on a filesystem at realistic utilization before committing.