Prometheus TSDB snapshot restore: no built-in way to confirm recovered samples match the original
0 reputation · 11 Mar 2021, 17:30 UTC
0 reputation · 11 Mar 2021, 17:30 UTC
We run Prometheus 2.x and back up TSDB snapshots plus archived WAL segments for disaster recovery. The documented snapshot and WAL replay mechanisms let us restore a point-in-time data directory, but I cannot find any supported way to verify that the restored data actually matches what was in the original instance.
The specific uncertainty: after replaying a snapshot (and optionally WAL segments on top), Prometheus starts and serves queries, but there is no native checksum, sample count, or integrity report confirming completeness. My concern is silent gaps — for example WAL segments that were archived mid-write, or out-of-order/duplicated samples introduced during replay — that would only surface later as subtly wrong query results.
Constraints: the restore target is a fresh instance on the same Prometheus version, and remote write replay to a second backend is also in scope, so duplicates there are a separate worry since remote write has no deduplication sequence numbers.
Is comparing identical PromQL query results and per-metric sample counts between a pre-restore baseline and the restored instance the accepted verification approach, or is there a more rigorous documented method? Should promtool tsdb analyze output be compared before and after, and which of its metrics actually indicate corruption versus normal variance? How do others detect silent WAL replay gaps without a known-good reference instance?
A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.