Question
Prometheus storage.tsdb.retention.time, scrape_interval and remote_write for low-traffic cost control
Mira Atlas
0 reputation · 01 Jan 2026, 04:58 UTC
21K views
Loading…
0 reputation · 01 Jan 2026, 04:58 UTC
Goal is to lower storage and resource consumption for a low-traffic Prometheus deployment by keeping only a short local window and offloading longer-term data.
The configuration involves --storage.tsdb.retention.time for local pruning, a coarser scrape_interval to reduce head block series churn, and remote_write to a separate store. Drop relabeling is also under consideration to remove high-cardinality labels at ingestion.
There is uncertainty about the interaction between local retention expiry and remote write delivery guarantees, and about the acceptable data granularity loss from a larger scrape interval for infrequent targets.
What is the effective data availability window when local retention is shorter than remote write buffering? How does scrape_interval interact with retention pruning for low-traffic series? Is there a documented permission boundary for drop relabeling that prevents accidental removal of metrics needed for remote write?
In a Prometheus deployment using remote_write, the local TSDB retention and the remote store's retention are independent. The --storage.tsdb.retention.time flag controls when local blocks are deleted from disk; it does not trigger deletions in the remote store.
The effective data availability window is split by location:
prometheus--storage.tsdb.retention.time.Risk: If local retention is set shorter than the time it takes for remote_write to successfully deliver and acknowledge samples during a network partition or remote outage, data may be pruned locally before it ever reaches the remote store. Because remote_write sends data from the head block, a very aggressive local retention period increases the risk of permanent data loss during prolonged remote-store downtime.
Increasing the scrape_interval reduces the total number of samples generated. For low-traffic series, this directly lowers the disk I/O and the number of samples stored in the head block before they are compacted into blocks.
While a coarser interval reduces storage costs, it introduces granularity loss. In low-traffic environments, short-lived spikes or transient errors that occur between scrapes will be missed entirely. This is a trade-off between cost and observability precision.
There is no separate "permission boundary" for drop relabeling. Relabeling occurs at the ingestion stage, before the data is written to the local TSDB or passed to the remote_write pipeline.
If a metric is dropped via relabel_configs, it is discarded globally. It will not be available for local queries, nor will it be sent to the remote store. To drop labels for local storage but keep them for remote storage, you must use remote_write specific relabeling configurations (remote_write
relabel_configs
) rather than global relabel_configs.
remote_write specific relabeling to drop high-cardinality labels only for the remote store, or use global relabeling to drop them everywhere.--storage.tsdb.retention.time is long enough to cover your maximum expected remote-store outage window (e.g., 24-48 hours).scrape_interval to the maximum acceptable limit for your SLIs.Verification: Use the following metric to monitor the number of series in the head block to verify the impact of your scrape interval and relabeling changes:
prometheus_tsdb_head_seriesNote: To provide a more precise retention recommendation, please specify the average uptime/reliability of your remote storage endpoint.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.