Choosing RocksDB Compaction Style for Write-Heavy Workloads: Level vs Universal

If your application is ingesting data faster than your SSD can absorb, or if p99 read latency spikes whenever compaction runs, the compaction style you select at database creation time is the primary lever. RocksDB offers two supported styles for this decision: level compaction and universal compaction. The useful takeaway: choose level compaction when point lookups dominate and you can tolerate higher write amplification; choose universal compaction when bulk ingest dominates and you need to minimize bytes written to storage, accepting slower random reads.

This guide frames the decision under three common constraints: a high ingest rate that fills memtables quickly, limited SSD endurance measured in terabytes written (TBW), and a requirement for predictable latency rather than throughput alone.

Decision constraints

Before comparing styles, confirm that your workload actually fits the write-heavy profile. The constraints that push the decision toward universal compaction are:

• Sustained ingest rate: Continuous writes that keep the write buffer (memtable) flushing to L0 many times per minute.
• Storage endurance limit: An SSD with a finite write budget where every extra byte written by compaction shortens the device's usable life.
• Bulk or range-scan reads: Access patterns that scan large key ranges rather than fetching individual keys.

If your workload is dominated by random point lookups—for example, a key-value cache or session store—level compaction is usually the safer default even under moderate write pressure.

Level vs universal compaction at a glance

The two styles organize sorted string tables (SSTs) differently. Level compaction arranges files into a hierarchy of non-overlapping levels (L0, L1, L2, …). Universal compaction keeps all files in a single tier of overlapping sorted runs. The practical differences are summarized below.

Aspect	Level compaction	Universal compaction
Write amplification	Higher; data is rewritten each time it moves down a level	Lower; fewer, larger merge jobs postpone rewrites
Read amplification / point latency	Lower; sorted levels reduce files checked per lookup	Higher; overlapping runs require more comparisons
Memory overhead	Moderate; level metadata and block cache pressure	Low; single-tier bookkeeping
Best fit	Mixed workloads, point queries	Write-heavy, bulk ingest, range scans

Write amplification is the ratio of physical bytes written to storage versus logical bytes written by your application. A ratio of 10 means every 1 MB of user data causes 10 MB of SSD writes. Read amplification counts how many SST files or blocks must be consulted to satisfy one logical read.

Trade-offs in practice

Level compaction keeps data sorted across levels, which minimizes the number of files a point query must inspect. The cost is that each compaction rewrites the entire key range, so a hot key can be copied through several levels, multiplying physical writes. On an endurance-limited SSD, that multiplier directly consumes TBW budget.

Universal compaction reduces write amplification by merging larger runs less frequently. Instead of promoting files level by level, it selects a group of sorted runs and compacts them into one larger run when a size threshold is reached. The trade-off is that multiple overlapping runs can exist simultaneously, so a random Get may need to binary-search several runs before finding the key. Range scans also pay a merge-sort cost across runs.

Latency predictability differs as well. Level compaction produces many small background jobs that can cause steady, moderate interference. Universal compaction produces fewer but larger jobs; when they do run, they can saturate disk I/O for longer periods, creating latency spikes if not throttled.

Concrete implementation: switching to universal compaction

The following C++ snippet configures a new database for universal compaction with a 256 MB write buffer. Run this in your application's initialization path, not in a shell. The process must have write permission to the database directory.

rocksdb::Options opts;
opts.compaction_style = rocksdb::kUniversalCompaction;
opts.write_buffer_size = 256 << 20;  // 256 MB
rocksdb::DB* db;
rocksdb::Status s = rocksdb::DB::Open(opts, "/var/lib/myapp/db", &db);
if (!s.ok()) {
  // handle error; do not proceed with a null db pointer
}

Placeholder /var/lib/myapp/db should be replaced with your actual database path. A larger write buffer reduces flush frequency but increases recovery time after a crash. If your ingest rate is extremely high, you may also raise max_write_buffer_number to allow more memtables to accumulate before stalling writes.

Validating the choice with db_bench

Before deploying, compare both styles under your own access pattern using the bundled benchmark tool. Run db_bench from your RocksDB build output directory. It needs write access to a temporary directory.

# Level compaction baseline
./db_bench --benchmarks=fillrandom,readrandom \
  --compaction_style=level \
  --db=/tmp/bench_level \
  --value_size=1024 --num=1000000

# Universal compaction candidate
./db_bench --benchmarks=fillrandom,readrandom \
  --compaction_style=universal \
  --db=/tmp/bench_universal \
  --value_size=1024 --num=1000000

Expected checks: compare the bytes written statistic reported by db_bench for each run. Universal should write fewer total bytes during the fill phase. Then compare micros per op for the readrandom phase; universal will typically show higher latency. If universal does not reduce bytes written, your size ratio or trigger settings may need tuning.

Inspecting runtime properties

After a production or staging run, query the database properties to confirm compaction behavior. In C++:

uint64_t bytes_written = 0;
db->GetProperty("rocksdb.compaction.bytes.written", &bytes_written);

Compare this value against the logical bytes your application wrote. A ratio close to 1 indicates minimal compaction overhead. Also inspect the LOG file in the database directory: universal compaction should show fewer, larger compaction events than level compaction. You can also check rocksdb.num-files-at-level0; under universal compaction, L0 may hold more files temporarily, while higher levels remain empty because data stays in a single tier.

Limitations and migration risk

The most important limitation is that compaction_style is not a mutable option. Changing it after a database has been created requires a full rebuild: either dump and reload the data through a new database, or use the offline ldb tool to rewrite SSTs. There is no supported in-place conversion.

Universal compaction also increases read amplification for point lookups. If your workload later shifts toward random reads, you may need to migrate back to level compaction. Test both styles with a representative read/write mix before committing to production.

Practical verification checklist

1. Run db_bench with both styles using your real value size and key distribution.
2. Confirm universal compaction lowers total bytes written during ingest.
3. Measure readrandom latency; ensure the increase is acceptable for your SLOs.
4. Open the LOG file and verify compaction job sizes match expectations.
5. If changing an existing database, plan a rebuild window—do not attempt to flip the style live.

If the SSD endurance constraint is absolute and your reads are predominantly range scans, universal compaction is the supported path. If random point reads are frequent and latency-sensitive, level compaction remains the lower-risk choice despite higher write amplification.