Backup restore versus abort-and-preserve: Prometheus upgrade recovery trade-offs

For production data continuity the safer default is abort-and-preserve with a binary rollback, not an immediate restore of a pre-upgrade backup. Stop the new binary, leave the current data directory untouched, restart the previous binary that was known to open the directory, and only consider a restore after you have verified version compatibility and quantified data loss.

Confirmed facts for this case

• Upgrades normally replace the binary and restart the process while the TSDB data directory remains in place.
• Prometheus aborts startup when blocks fail checksum validation. It does not delete or truncate invalid blocks on its own.
• The /-/reload endpoint reloads configuration only. It does not re-read or repair the data directory and cannot resolve a startup abort caused by block corruption.

Likely explanation, not confirmed

The risk assessment below assumes corruption is isolated to a small set of blocks and that a pre-upgrade backup is both complete and schema-compatible with the new binary. Those assumptions are not guaranteed. A backup can be stale, missing samples written after the backup point, and may be rejected by a newer binary if storage or schema changed between versions.

Minimal steps for this situation

1. Stop the new binary. Do not modify the live data directory.
2. Copy the current data directory aside for forensic use. Keep the original in place.
3. Start the previous Prometheus binary against the current data directory. Confirm it reaches ready state and serves queries for the expected time range.
4. Inspect blocks on a non-production copy with promtool tsdb analyze and block metadata inspection for your version. Identify which block directories fail.
5. If failures are isolated, quarantine suspect block directories by moving them, not deleting them, to a path outside the data directory. Restart the previous binary to confirm stability.
6. Only if the previous binary cannot start or the WAL/head is implicated, restore from a verified backup into a separate path. Compare block min/max times against the current directory to quantify the data gap before any cutover.

Weighing restore vs preserve

A non-starting process with intact blocks is recoverable by rolling back the binary. A restored but stale or schema-incompatible directory can silently lose recent samples and still fail to start, creating a larger outage. Restoring trades an outage for two hidden risks: loss of post-backup samples and possible incompatibility with the new binary.

One missing diagnostic changes the recommendation: are checksum failures confined to a few older block directories, or do they also involve the WAL/head? If only isolated blocks are bad, quarantine and keep the directory. If the WAL/head or many blocks fail, a verified backup restore into a separate path is the more defensible path.