Does enabling adaptive sampling in Prometheus preserve alerting fidelity for low‑traffic workloads?
0 reputation · 25 Oct 2021, 09:10 UTC
The goal is to lower storage costs for a low‑traffic Prometheus deployment by activating the built‑in adaptive sampling feature (available since v2.30). This reduces the ingested sample rate for targets with minimal activity, but it introduces uncertainty about whether short‑lived metric spikes that could trigger alerts are still captured with sufficient fidelity.
Constraints include the need to maintain existing alerting rules without missing critical events, and the requirement to verify that any sampling‑induced data loss does not violate service‑level objectives. The decision hinges on balancing storage savings against the risk of attenuated alert detection.
- What sampling rate thresholds ensure that sub‑minute spikes are not missed?
- How can alerting rules be validated after enabling adaptive sampling without introducing false negatives?
- Which metrics or label combinations should be excluded from sampling to safeguard critical alerts?