Does enabling adaptive sampling in Prometheus preserve alerting fidelity for low‑traffic workloads?
0 reputation · 25 Oct 2021, 09:10 UTC
0 reputation · 25 Oct 2021, 09:10 UTC
The goal is to lower storage costs for a low‑traffic Prometheus deployment by activating the built‑in adaptive sampling feature (available since v2.30). This reduces the ingested sample rate for targets with minimal activity, but it introduces uncertainty about whether short‑lived metric spikes that could trigger alerts are still captured with sufficient fidelity.
Constraints include the need to maintain existing alerting rules without missing critical events, and the requirement to verify that any sampling‑induced data loss does not violate service‑level objectives. The decision hinges on balancing storage savings against the risk of attenuated alert detection.
35275 reputation · 25 Oct 2021, 11:40 UTC
The premise needs correcting before the question can be answered: upstream Prometheus has no built-in "adaptive sampling" feature, in v2.30 or any later release. Prometheus scrapes each target at a fixed, configured interval and stores every sample it scrapes. There is no server-side mechanism that reduces the ingested sample rate for low-activity targets, so there is no such feature to enable — and therefore no sampling-induced fidelity loss to worry about from Prometheus core itself.
Because no live documentation was fetched for this answer, treat the version-specific claim with appropriate skepticism and verify against the official changelog — but the architectural point is well established: Prometheus's storage model is "store what you scrape," and cost control happens through scrape configuration and relabeling, not adaptive sampling.
Sampling-like behavior does exist elsewhere in the ecosystem, which may be the source of the mix-up:
You can confirm what your server actually supports by checking /api/v1/status/flags and your scrape configs — you will find only standard scrape and relabel settings, no sampling flag.
Alerting fidelity in a stock deployment is governed by two knobs, and both matter for your sub-minute spike concern:
for durations. Lengthening the scrape interval to save storage also slows alert evaluation and delays staleness handling.So the honest answer to "what sampling rate ensures sub-minute spikes aren't missed" is: no sampling rate does — only a scrape interval shorter than the spike duration does.
For a low-traffic deployment, the practical levers are:
metric_relabel_configs (e.g., high-cardinality metrics no alert references). This is irreversible, so scope the regex carefully.The safeguard for critical alerts is simple: any metric or label set referenced by an alerting rule must be excluded from drop rules and kept at the original scrape interval.
Before applying any drop config to production:
promtool test rules against recorded series to confirm each rule still has input data.One detail that would sharpen this recommendation: which backend (plain Prometheus, Thanos, Mimir, or an agent-based pipeline) is actually in front of your storage — because that determines whether any real sampling layer exists in your path at all.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.