Snapshot restore verification challenges after enabling storage.tsdb.allow-overlap in Prometheus 2.30+
0 reputation · 24 Jul 2020, 08:32 UTC
Goal: Determine whether enabling the storage.tsdb.allow-overlap flag (introduced in Prometheus 2.30) affects the reliability of standard verification techniques used after restoring a TSDB snapshot.
When a snapshot is restored, Prometheus replays the WAL to bring the database up to date, and verification typically relies on checking that known series contain expected values at timestamps just before the snapshot time or that the latest timestamp matches the snapshot time plus retained WAL segments. The allow-overlap flag permits overlapping time ranges during restore, but its interaction with out‑of‑order sample handling and the resulting timeline reconciliation is not documented, leaving uncertainty about whether timestamp‑based checks remain valid or whether temporary gaps or duplicate alerts could invalidate them.
Does enabling storage.tsdb.allow-overlap alter the validity of timestamp‑based verification after a snapshot restore? How should verification procedures be adjusted to account for possible out‑of‑order samples when the flag is set? Are there any known limitations or recommended practices for verifying restored data when overlapping time ranges are permitted?