Question
Snapshot restore verification challenges after enabling storage.tsdb.allow-overlap in Prometheus 2.30+
Zo Indigo
0 reputation · 24 Jul 2020, 08:32 UTC
106.2K views1
Loading…
0 reputation · 24 Jul 2020, 08:32 UTC
Goal: Determine whether enabling the storage.tsdb.allow-overlap flag (introduced in Prometheus 2.30) affects the reliability of standard verification techniques used after restoring a TSDB snapshot.
When a snapshot is restored, Prometheus replays the WAL to bring the database up to date, and verification typically relies on checking that known series contain expected values at timestamps just before the snapshot time or that the latest timestamp matches the snapshot time plus retained WAL segments. The allow-overlap flag permits overlapping time ranges during restore, but its interaction with out‑of‑order sample handling and the resulting timeline reconciliation is not documented, leaving uncertainty about whether timestamp‑based checks remain valid or whether temporary gaps or duplicate alerts could invalidate them.
Does enabling storage.tsdb.allow-overlap alter the validity of timestamp‑based verification after a snapshot restore? How should verification procedures be adjusted to account for possible out‑of‑order samples when the flag is set? Are there any known limitations or recommended practices for verifying restored data when overlapping time ranges are permitted?
35275 reputation · 24 Jul 2020, 09:33 UTC
Yes. Enabling storage.tsdb.allow-overlap weakens single-point timestamp checks after a TSDB snapshot restore, but it does not invalidate range-based verification. The flag relaxes block range validation so restored blocks may overlap existing WAL or head ranges; Prometheus can then replay out-of-order samples and reconcile timelines during compaction. A check that asserts latest timestamp equals snapshot time plus retained WAL can therefore flip-flop or fail while overlap exists.
Likely explanation, treat as unverified for your build: with the flag enabled, restored blocks can carry duplicate timestamps or samples older than the current head inside the overlap window. Query semantics and block format are expected not to change, but the meaning of latest timestamp weakens because newer restored samples may sort older than head samples until compaction merges or drops redundant ranges.
Expected behavior: allow-overlap is a validation relaxation, not a repair mechanism. It does not guarantee snapshot and WAL consistency, and it does not recover corrupted blocks. Prometheus 2.30+ minor releases may enforce the flag differently, so confirm behavior against your deployed version before production use.
Are you restoring into an empty data directory or over an existing WAL or head block? This determines whether overlap verification is required at all.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 24 Jul 2020, 19:18 UTC
Enabling storage.tsdb.allow-overlap lets Prometheus keep blocks that share time ranges. During a snapshot restore the WAL is replayed over the restored data, but if an overlapping block contains a sample with the same timestamp as a WAL entry, the engine will keep the value that was written last. This can make a simple "value at timestamp T" check ambiguous.
prometheus/api/v1/query?query=metric{...}&time=SNAPSHOT_TIME-1s) to read the value from the block that existed before the replay.block_meta.json files to confirm that overlapping blocks exist and to count duplicate series.These steps keep verification reliable while accounting for the potential duplicate samples introduced by the flag.