Architecture Note: Using RocksDB Leveled Compaction for Write‑Heavy Workloads

Requirements

The system must sustain high write throughput while keeping read latency predictable. Data durability across process crashes and power loss is required, and the storage medium is assumed to be an SSD with known endurance limits.

Smallest Suitable Design

Enable the write‑ahead log (WAL) with synchronous writes, size the memtable to fit comfortably in memory, and select leveled compaction because it bounds read amplification and keeps read latency stable under steady writes.

// Example RocksDB options (C++ API)
rocksdb::Options options;
options.create_if_missing = true;
options.WAL = true;
options.WAL_ttl_seconds = 0; // retain WAL until file is no longer needed
options.WAL_size_limit_MB = 0; // no artificial size limit
options.sync = true; // fsync after each write batch
options.max_write_buffer_number = 2; // allow one immutable memtable
options.write_buffer_size = 64 << 20; // 64 MiB memtable
options.compaction_style = rocksdb::kLeveledCompaction;
options.target_file_size_base = 8 << 20; // 8 MiB base size
options.max_bytes_for_level_base = 64 << 20; // 64 MiB for L0
options.max_bytes_for_level_multiplier = 10; // each level ~10× previous
options.level0_file_num_compaction_trigger = 4; // modest trigger
options.level0_slowdown_writes_trigger = 8; // begin slowing writes
options.level0_stop_writes_trigger = 12; // hard stop
// Encryption example (if confidentiality needed)
// options.enable_encryption = true;
// options.encryption_password = "my‑strong‑password";
// options.encryption_provider = rocksdb::EncryptionProvider::CreateAESProvider();

This configuration represents the minimal set that satisfies the durability and write‑throughput goals while keeping the design simple enough to reason about.

Trust and Data Boundaries

• WAL and SST files are treated as trusted storage; only the RocksDB process should have read/write access.
• Isolate the RocksDB directory via filesystem permissions (e.g., chmod 700 /var/rocksdb) and run the database under a dedicated unprivileged user.
• If data confidentiality is required, enable RocksDB’s built‑in encryption with a key managed outside the process (e.g., via a hardware security module or a key‑management service). Encryption does not alter the compaction logic but ensures that raw SST files are unreadable without the key.

Operational Checks

Monitor the following metrics, preferably exported via RocksDB’s statistics interface or a monitoring agent:

• Write stall signals – rocksdb.write.stall.micros; alert if the 95th‑percentile exceeds 100 ms.
• WAL sync latency – rocksdb.wal.sync.micros; sustained high values indicate storage backend issues.
• Compaction pressure – rocksdb.compaction.pending.bytes and rocksdb.compaction.running; a growing pending‑bytes trend suggests the compaction queue is falling behind.
• Disk I/O utilization – from the OS (e.g., iostat); watch for >80 % utilization on the device hosting the RocksDB directory.

Example check using the RocksDB C++ API:

std::string stall_micros;
db->GetProperty("rocksdb.write.stall.micros", &stall_micros);
if (std::stod(stall_micros) > 100000) { // 100 ms in microseconds
    // trigger alert or auto‑tune
}

Failure Modes

• WAL sync failure – if the underlying filesystem cannot fsync, RocksDB returns an error and the latest batch may be lost. Recovery relies on restoring from a backup or a replicated WAL.
• Compaction lag – when compaction cannot keep pace with incoming writes, level0 files accumulate, triggering write slowdowns and eventually a write stall. Prolonged lag can exhaust memory as immutable memtables pile up, risking OOM.
• Disk full – RocksDB returns STATUS_IO_ERROR and switches to read‑only mode; existing SST files remain intact, but new writes are rejected until space is freed.
• Power loss – with sync=true the WAL is flushed to durable storage; on restart RocksDB replays the WAL and restores the memtable state, guaranteeing that committed writes are recovered.

Conditions That Would Change the Design

• Read‑heavy workload – leveled compaction’s write amplification becomes a liability. Switching to universal compaction or increasing the block cache size can improve read latency at the cost of higher write amplification.
• Sub‑millisecond read latency requirement – reduce memtable size (e.g., to 32 MiB) to decrease flush frequency and increase the block cache to keep hot data in memory.
• Regulatory encryption mandate – enable RocksDB encryption with hardware‑accelerated AES (AES‑NI) to meet confidentiality requirements without a major redesign.
• SSD endurance concerns – if the expected write amplification from leveled compaction exceeds the SSD’s rated TBW, consider a tiered compaction style or a larger target file size to reduce amplification, accepting higher read amplification.

Verification Steps (for a test environment)

1. Create a temporary directory, initialize a RocksDB instance with the options above.
2. Drive a steady write workload (e.g., 100 k ops/sec of 1 KiB random key‑value pairs) using a benchmark tool such as db_bench.
3. Observe the rocksdb.write.stall.micros metric; verify it stays below the 100 ms threshold for the duration of the test.
4. Simulate a power loss by sending SIGKILL to the RocksDB process, then restart and confirm that the number of keys written before the kill matches the number readable after recovery.
5. Fill the filesystem to 95 % capacity and verify that write operations return STATUS_IO_ERROR while existing SST files remain readable.
6. If encryption is enabled, stop the process, attempt to dump an SST file with sst_dump, and confirm the output is unintelligible without the key.