When Write Amplification Hurts: Using RocksDB Universal Compaction for Write-Heavy Stores

Write-heavy services hit a wall when background compaction becomes the bottleneck. With RocksDB's default leveled compaction, each key can be rewritten many times as it moves down levels, inflating write amplification and I/O. The useful takeaway is that universal compaction trades some read efficiency for much lower write amplification by merging many small SST files into fewer large ones, a good fit for append-heavy or time-series workloads.

The problem with leveled compaction under sustained writes

Leveled compaction keeps data in levels with size targets, guaranteeing low read amplification for point lookups. The cost is write amplification: a key written to the memtable is flushed to L0, then repeatedly copied during level-to-level compactions.

For workloads that ingest a lot of new keys with few updates, that copying is largely wasted work. Background I/O grows, write latency becomes noisy, and SSD endurance is consumed faster.

How universal compaction changes the merge pattern

Universal compaction, available from RocksDB 6.5 onward, does not enforce strict level sizes. It accumulates SST files and periodically merges a set of smaller files into one larger file.

Because merges are size-based and skip intermediate levels, each byte is rewritten fewer times. Write amplification drops. Point lookups stay fast because the Bloom filter and index are still per file. Range scans can suffer because a scan may need to touch more data spread across larger files, and read amplification can increase.

Configuring universal compaction safely

Set the style at open time. The option is immutable for an existing DB; changing it requires a new database or a dump/reload.

// C++ example, run where you open the DB with write permission to <db_path>
rocksdb::Options opts;
opts.create_if_missing = true;
opts.compaction_style = rocksdb::CompactionStyle::kCompactionStyleUniversal;

// Tuning knobs that matter
opts.universal_compaction_options.size_ratio = 2;          // merge when next file is at least 2x larger
opts.universal_compaction_options.max_size_amplification_percent = 200;
opts.universal_compaction_options.min_merge_width = 2;
opts.universal_compaction_options.max_merge_width = 10;
opts.universal_compaction_options.max_bytes_for_level_base = 64 * 1024 * 1024; // 64 MiB base

rocksdb::DB* db;
rocksdb::Status s = rocksdb::DB::Open(opts, "/var/lib/rocksdb/universal_example", &db);

Meaningful placeholders: replace the path with a directory you own and can write to. Required permissions are read/write on that directory and its files. Risk: an overly small max_bytes_for_level_base can cause many levels and higher memory use for file metadata; an overly large value can create very large merge jobs.

Verify the style is active after open:

std::string value;
db->GetProperty("rocksdb.compaction-style", &value);
// Expect value to indicate universal

Check compaction behavior with level bytes:

uint64_t bytes;
db->GetProperty("rocksdb.cur-size-all-levels", &bytes);

Observing CURRENT_LEVEL_BYTES before and after a batch of inserts shows how files are grouped rather than strictly leveled.

Worked example for a write-heavy ingestion pattern

Illustrative flow for a new DB with universal compaction:

• Create the DB with the options above.
• Write a batch of monotonically increasing keys, e.g., 1M key-value pairs, using WriteBatch.
• Force a flush to move data from memtable to SST files.
• Read back a few keys to confirm visibility.
• Query db->GetProperty for compaction style and size metrics to confirm the DB is using universal compaction and that files are merging.

This pattern demonstrates the configuration path without claiming the example was tested in your environment.

Trade-offs and limitations

Universal compaction reduces write amplification but can increase read latency for range scans because larger files mean more data to skip. It also changes space amplification behavior; deleted keys are not removed as aggressively as in leveled compaction.

Tuning max_bytes_for_level_base is critical. Too low creates many levels and metadata overhead. Too high creates huge merge jobs that pause writes. The feature is not available in RocksDB versions earlier than 6.5.

A practical way to check the result is to compare compaction metrics between leveled and universal on a representative workload, for example YCSB write-heavy, and monitor write amplification and P99 write latency. Keep the DB on a filesystem with adequate free space for merge outputs.

Use universal compaction when writes dominate, updates are rare, and point lookups matter more than fast range scans. For mixed or read-heavy workloads, leveled compaction often remains the safer default.