Prometheus TSDB snapshot restore: no built-in way to confirm recovered samples match the original

Prometheus does not provide a native checksum, manifest, or built-in integrity report to confirm that a restored TSDB directory exactly matches the original source. Because the snapshot API creates hard links to block files and the WAL replay is a deterministic but silent process, there is no internal mechanism to detect "silent gaps" caused by partial WAL writes or missing segments during restoration.

Verification Strategy

The accepted industry approach for verifying a restore is comparative auditing. Since there is no pre-calculated hash for the entire database, you must establish a baseline from the source and compare it against the restored instance.

1. Structural Integrity Check

Before checking for data gaps, verify that the on-disk files are not corrupted. Use promtool to validate the structural integrity of the blocks (index/chunk cross-references and magic bytes).

promtool tsdb check /path/to/restored/data

A non-zero exit code indicates structural corruption, meaning the restore failed at the filesystem or block level.

2. Logical Data Comparison

To detect missing samples or WAL gaps, run identical PromQL queries against both the original (if available) and the restored instance. Focus on aggregate counts and time ranges:

• Series Count: count({__name__=~".+"})
• Sample Distribution: count_over_time({__name__=~"your_critical_metric".}[24h])
• Counter Totals: Compare the sum() of critical counters over the restoration window.

3. Using promtool tsdb analyze

You can use promtool tsdb analyze to get a high-level overview of the database. However, do not expect exact equality in sample counts if compaction has run on one instance but not the other. To get a meaningful comparison:

1. Disable compaction on both instances (set --storage.tsdb.retention.time=0 and --storage.tsdb.min-block-duration=0) to freeze the state.
2. Run promtool tsdb analyze on both directories.
3. Compare the Series Count and Block Count. If the restored instance has fewer blocks or a significantly lower series count, you have a gap in your snapshot or WAL archive.

Handling WAL Gaps and Remote Write

If you lack a reference instance, the only way to detect non-deterministic WAL replay is to perform the restore twice into two separate fresh instances. If the resulting query outputs diverge, your WAL archive contains corruption or torn writes.

Regarding Remote Write: Because Prometheus remote write lacks per-sample sequence IDs, replaying data to a second backend will inevitably create duplicates. This is a known limitation; deduplication must be handled by the receiving backend (e.g., Thanos Compactor) using last-write-wins semantics.

Required Diagnostic Detail

To refine this recommendation: Are you archiving WAL segments via the Prometheus snapshot API, or via manual filesystem copies of the wal/ directory? Manual copies are prone to "torn writes" (partial records), which promtool tsdb check may not detect but will cause sample gaps.