Write Stall Mode: kStopWritesFirst vs kSlowdown for p99 Latency Under Concurrent Writers

To bound p99 write latency under concurrent ingest, choosing between kStopWritesFirst and kSlowdown depends on whether your application prioritizes immediate request rejection or gradual throttling.

For a latency-sensitive service that can retry quickly, kStopWritesFirst typically produces a lower p99 latency for accepted writes. By immediately rejecting writes with Status::TryAgain once the stall limit is hit, it prevents the 'convoy effect' where multiple threads block on a saturated engine. In contrast, kSlowdown uses a rate limiter to throttle writers, which preserves throughput but allows the p99 latency to stretch as threads wait for the limiter to drain the queue.

Key Differences in Backpressure

• kStopWritesFirst (Hard Limit): This acts as a circuit breaker. It provides a hard latency bound for the requests that succeed, but shifts the burden of latency to the client-side. This is ideal if your client-side logic is optimized for rapid, asynchronous retries.
• kSlowdown (Soft Limit): This engages a gradual backpressure mechanism. It avoids sudden spikes by slowing down the ingestion rate. While this maintains sustained throughput, the tail latency (p99) will increase because threads are held in a waiting state longer longer.

Tuning and Compaction Debt

When switching between modes, the relationship between thresholds and rate limits is critical:

• write_stall_limit: In kStopWritesFirst, keep this lower to trigger early rejection and protect latency. In kSlowdown, this limit acts as a safety net; the rate limiter should be the primary driver.
• bytes_per_sync and delayed_write_rate: These should be tuned to match your I/O subsystem's sustained compaction capacity. If the rate limit is too high, kSlowdown will eventually hit the hard stall limit anyway, defeating the purpose.

Regarding **compaction debt, kSlowdown generally results in more stable debt accumulation because it prevents the write rate from vastly exceeding the compaction threads' capacity. kStopWritesFirst can lead to fluctuations in debt if the bursty ingest is followed by total stalls, potentially impacting subsequent read latency when the engine catches up.

To provide a more specific tuning recommendation, is your client-side retry logic capable of handling high-frequency rejections, or is it synchronous blocking?