Question
RocksDB lazy format upgrade one-way MANIFEST rewrite and recovery options
Ira Atlas
0 reputation · 12 Jan 2023, 19:20 UTC
views
Loading…
0 reputation · 12 Jan 2023, 19:20 UTC
RocksDB performs format upgrades lazily on open when a newer library version implies a newer on-disk layout. The upgrade rewrites MANIFEST and SST metadata rather than requiring an explicit upgrade command, and the change is effectively one-way. Once the MANIFEST is rewritten for a newer format, older RocksDB binaries typically cannot open the database, so downgrade is not supported by design.
There is no built-in automatic rollback mechanism for a failed or interrupted upgrade. Safe recovery therefore depends on operator-provided safeguards such as pre-upgrade backups, checkpoints, or retaining the previous binary version. The operational decision of whether and how to preserve a pre-upgrade MANIFEST snapshot automatically remains unresolved, and the exact state left after interruption is version-sensitive and may interact with WAL recovery and compaction state.
Is a pre-upgrade MANIFEST snapshot preserved automatically by RocksDB during a lazy format upgrade? If the upgrade is interrupted during MANIFEST rewrite, what state is the database left in for the same version that initiated the upgrade? What operator safeguards are considered sufficient for safe recovery given the lack of built-in rollback?
33275 reputation · 13 Jan 2023, 00:15 UTC
RocksDB does **not** automatically keep a copy of the pre‑upgrade MANIFEST during a lazy format upgrade. The upgrade rewrites the existing MANIFEST file in place, so after a successful rewrite the old MANIFEST is lost.
If the rewrite is interrupted (process killed, power loss, etc.) the MANIFEST can be left partially written or corrupted. When the same RocksDB version that started the upgrade tries to reopen the database, the open operation fails with a corruption or I/O error – the database is not usable until the MANIFEST is restored.
rocksdbcp -r db db.preupgrade).rocksdb::DB::Open returns Status::OK()).If you need to distinguish between a partial MANIFEST rewrite that still contains a valid header versus total corruption, request the exact error code or ldb dump --manifest output from the failed open attempt; that will determine whether a manual MANIFEST repair is feasible or a full restore is required.
Use comments to ask for clarification. Post a solution as an answer.
33,275 reputation · 12 Jan 2023, 23:23 UTC
During a lazy format upgrade the engine first creates a fresh MANIFEST-xxxxx file that contains the upgraded metadata. Once the file is fully written it is atomically renamed to replace the old MANIFEST. The previous manifest is only deleted during a later checkpoint or compaction.
If the process is killed after the rename but before compaction finishes, the database opens with the new manifest while some SST files still use the old format. Operations that touch those SSTs may fail until the compaction reconciles the formats.
Operators can rely on keeping the old binary available and on a pre‑upgrade backup (filesystem snapshot or BackupEngine) to restore the database to the old state. The old manifest remains safe until it is purged by compaction, giving a window for recovery.