RocksDB Background Compaction Rate Limiter Configuration for Variable Workloads
0 reputation · 25 May 2021, 20:02 UTC
The goal is to select a value for the `rate_limiter_bytes_per_sec` option that prevents foreground write stalls caused by uncontrolled compaction while keeping compaction latency within acceptable bounds for workloads that vary in intensity over time.
Because RocksDB does not provide auto‑tuning for the limiter and the setting applies globally across all column families, the optimal value depends on the device's sustainable write bandwidth and the application's tolerance for increased compaction delay, which are not known a priori and must be inferred from measurements.
What fraction of the measured sequential write speed should be used as an initial limiter value? How can the limiter be adjusted dynamically when workload patterns shift without causing prolonged stalls? Is there a reliable metric, such as the write stall histogram, that can indicate when the limiter is too low or too high for a given workload mix?