Question
scrape_timeout equal to scrape_interval is accepted by Prometheus 2.x config validation
Sky Indigo
0 reputation · 03 Apr 2023, 11:42 UTC
38.4K views1
Loading…
0 reputation · 03 Apr 2023, 11:42 UTC
Prometheus documentation recommends that scrape_timeout stay shorter than scrape_interval for each scrape config. The configuration parser does not hard-reject values where scrape_timeout equals or exceeds scrape_interval, so overlapping scrapes against the same target can occur.
When a scrape exceeds its timeout, Prometheus cancels the HTTP request through Go context and discards the incomplete sample set, but the exporter may continue processing after disconnection. A single timeout does not immediately mark the target down; target health depends on consecutive failures across scrape_interval cycles.
Is this overlap behavior intentional for backward compatibility, or should operators treat scrape_timeout greater than or equal to scrape_interval as a misconfiguration? Does Prometheus track or limit concurrent scrapes per target when intervals overlap? Should configuration validation warn or reject overlapping scrape windows?
35275 reputation · 03 Apr 2023, 19:38 UTC
Prometheus 2.x’s configuration parser does not reject a scrape_timeout that is equal to or longer than the scrape_interval. The parser only checks that the timeout is a positive duration, so overlapping scrape windows are allowed. This permissiveness is retained for backward compatibility with older clusters that relied on the same rule.
prometheusscrape_timeout is positive; it does not warn or reject overlapping windows.Overlapping scrapes may increase load on the exporter, cause duplicate or missing samples, and make troubleshooting more difficult. Operators should therefore configure scrape_timeout to be strictly less than scrape_interval unless a specific use case requires otherwise.
scrape_timeout equal to scrape_interval.
/api/v1/debug/scrape to see the number of in‑flight scrapes per target. Overlaps will appear as multiple concurrent entries.
Treat scrape_timeout >= scrape_interval as a misconfiguration for production deployments. Set scrape_timeout to a value smaller than scrape_interval to avoid overlapping scrapes and the associated risks.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 03 Apr 2023, 16:27 UTC
In Prometheus 2.x the promtool check config validator rejects any scrape_timeout that is strictly greater than scrape_interval. Equality is permitted because the check uses a > comparison, not >=. So a config with scrape_timeout: 30s and scrape_interval: 30s passes validation, but scrape_timeout: 31s fails.
scrape_timeout equals scrape_interval, the request deadline and the next scheduled tick coincide. After the context is cancelled, the next scrape can begin almost immediately, which may look like overlap from the exporter’s perspective.up=0 for that scrape, but the target is not marked down until alerting rules aggregate consecutive failures.Operators should keep scrape_timeout strictly below scrape_interval to preserve a safety margin and avoid back‑to‑back scrapes that can stress exporters.