SST file format incompatibility during RocksDB version downgrade

SST Format Compatibility

A minor version update in RocksDB does not always maintain the same SST file format version. While backward compatibility (new versions reading old files) is a core guarantee, forward compatibility is not. If a minor version introduces an optimization to the SST footer or block format, any file rewritten by that version will be unreadable by previous releases.

There is no fixed threshold of version increments (e.g., "every three versions") that triggers incompatibility. Instead, incompatibility occurs the moment a release increments the internal on-disk format version. This can happen in any release, including minor updates, if the storage engine's internal layout is modified.

Likely Cause of Downgrade Failure

When you upgrade RocksDB, the existing SST files remain in the old format until they are touched. The incompatibility typically triggers after one of the following events occurs under the newer version:

• Compaction: The engine merges multiple SST files into a new one using the current version's format.
• Ingestion: New data is written to disk using the current version's format.
• Flush: Memtables are flushed to L0 SST files using the current version's format.

If you attempt to downgrade after these events, the older binary encounters a file footer version it does not recognize and fails to open the database to prevent data corruption.

Recovery and Verification Steps

If you must revert to a previous version, follow these scoped steps:

1. Restore from Backup: The only guaranteed recovery method is restoring a snapshot or backup taken before the upgrade and subsequent compactions.
2. Verify Format Versions: Use the sst_dump utility (provided with RocksDB) to inspect the footer of your SST files. Compare the format version of a file created by the old version versus one created by the new version.
3. Controlled Testing: Before deploying an upgrade, trigger a manual compaction in a staging environment: db->CompactRange(column_family, NULL, NULL). Attempt to open that specific directory with the previous binary to verify if the format changed.

Required Diagnostic: To provide a more specific compatibility window, please provide the exact version numbers (e.g., v7.1.0 to v7.2.0) you are transitioning between.