Remote Write Queue Memory Limits and Sample Dropping

Remote Write Queue Behavior

Prometheus prioritizes the most recent data when the remote write queue reaches its capacity. It employs a First-In-First-Out (FIFO) dropping strategy: when the queue is full, the oldest samples are discarded to make room for incoming new samples.

Memory Impact of High-Cardinality Churn

High-cardinality churn—where the number of unique time series increases rapidly due to ephemeral labels—significantly increases the memory footprint of the remote write buffer. This happens because the buffer must track metadata for each unique series it is attempting to transmit. Even if the total number of samples remains constant, a high volume of unique series increases the overhead per sample, accelerating the approach toward the memory limit and increasing the likelihood of OOM (Out-Of-Memory) events.

Operational Guidance

To manage the balance between data retention and system stability, follow these steps:

1. Quantify Data Loss: Monitor the prometheus_remote_storage_samples_dropped_total metric. A rising counter indicates that your ingestion rate is exceeding your transmission capacity.
2. Analyze Queue Depth: Check prometheus_remote_storage_queue_length. If this value consistently hits your configured capacity limit, the queue is undersized for your current latency/throughput profile.
3. Adjust Capacity Carefully: If you increase the queue capacity to mitigate drops, you must proportionally increase the available system RAM. Failure to do so may result in the kernel killing the Prometheus process (OOM kill) rather than the application gracefully dropping samples.

Verification Commands

You can verify the current state of your remote write pipeline using the Prometheus HTTP API:

# Check for dropped samples via the query API
curl -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=prometheus_remote_storage_samples_dropped_total'

# Check current queue length
curl -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=prometheus_remote_storage_queue_length'

Note: This analysis assumes a standard Prometheus deployment. If you are using a specific distribution or a sidecar proxy for remote write, the buffering behavior may differ based on the proxy's own memory management.

To provide a more specific recommendation on capacity tuning: are you seeing a steady climb in dropped samples or sharp spikes coinciding with specific events?