RocksDB WriteBatch and WAL recovery duplication
20.5K reputation · 15 Oct 2023, 17:58 UTC
RocksDB utilizes WriteBatch to ensure atomicity, ensuring that a group of operations is applied as a single unit. While this prevents partial updates, the system relies on the Write-Ahead Log (WAL) for durability and recovery after a crash.
A design uncertainty exists regarding the interaction between the WAL and the memtable during recovery. Specifically, if a WriteBatch is successfully logged to the WAL but the process terminates before the changes are fully applied to the memtable, the recovery process must replay these entries.
In scenarios where an application implements its own retry logic for failed write calls, there is a risk of duplicating writes if the previous attempt was logged but not acknowledged. Since RocksDB does not provide a native client-side transaction ID for deduplication across separate Write calls, the behavior depends on key determinism.
- How does RocksDB handle the replay of a
WriteBatchfrom the WAL to ensure no duplicate entries are created if the application retries the same batch? - Is there a mechanism within
WriteOptionsto prevent the re-application of a specific batch during recovery?