Remote Write Queue Memory Limits and Sample Dropping
0 reputation · 23 Mar 2025, 07:00 UTC
Remote Write Buffer Management
Prometheus utilizes an asynchronous remote write queue to buffer samples before they are transmitted to a remote storage backend. This mechanism decouples local ingestion from the latency of the remote destination to ensure the stability of the local TSDB.
Queue Capacity and Stability
The remote write queue is managed by configuration parameters that define the maximum number of samples held in memory. When the ingestion rate exceeds the transmission rate and the queue reaches its capacity, Prometheus is designed to drop samples to prevent out-of-memory (OOM) crashes.
In environments with high-cardinality data, there is a tension between maximizing data retention during network spikes and maintaining system stability. Increasing the queue limits may mitigate data loss but increases the risk of OOM kills if system memory is constrained.
- How does Prometheus prioritize which samples are dropped when the remote write queue is full?
- What is the impact of high-cardinality churn on the memory footprint of the remote write buffer?
prometheus