Write Stall Mode: kStopWritesFirst vs kSlowdown for p99 Latency Under Concurrent Writers

Goal

Bound p99 write latency when multiple threads ingest into RocksDB concurrently, without sacrificing sustained throughput.

Constraints and Uncertainty

RocksDB triggers a write stall when memtable usage, L0 file count, or compaction debt exceed internal thresholds. Under concurrent writers this stall manifests as sudden p99 spikes. Two documented stall modes offer different backpressure semantics:

• kStopWritesFirst — immediately rejects writes with Status::TryAgain once the stall limit is reached, providing hard latency bounds at the cost of client-side retries.
• kSlowdown — engages the rate limiter to throttle writers gradually, preserving throughput but allowing tail latency to stretch while the limiter drains.

The optimal choice depends on workload burstiness, retry logic availability, and whether the service can tolerate rejected requests versus elongated pauses. Thresholds (write_stall_limit, max_write_buffer_number, level0_stop_writes_trigger) interact with the selected mode in ways that are not fully deterministic across hardware and I/O schedulers.

Questions

1. For a latency-sensitive service that can retry quickly, does kStopWritesFirst consistently produce lower p99 than kSlowdown under sustained concurrent ingest?
2. How should write_stall_limit be tuned relative to the rate limiter’s bytes_per_sync and delayed_write_rate when switching between modes?
3. Are there observable differences in compaction debt accumulation between the two modes that affect subsequent read latency?