Backup restore versus abort-and-preserve: Prometheus upgrade recovery trade-offs

In current Prometheus releases, upgrades typically replace the binary and restart the process while the data directory remains unchanged, preserving existing time-series blocks. When the new binary aborts during startup because blocks fail checksum validation, operators face a choice between restoring a pre-upgrade backup of the data directory or accepting the current abort-and-preserve behavior that leaves the directory untouched.

Restoring a backup preserves the original data, but a schema change between versions can leave the restored directory incompatible with the new binary. The /-/reload endpoint reloads configuration only and does not re-read or repair the data directory, so it cannot resolve startup aborts caused by block corruption. Current behavior aborts startup rather than deleting or truncating invalid blocks.

Given the unresolved question of whether Prometheus should provide a built-in repair mode that skips or rebuilds corrupted blocks, which recovery strategy offers the safer trade-off for production data continuity? How should operators weigh the risk of stale or incompatible restored data against the operational cost of a non-starting process when choosing between these two approaches?