Remote Write Queue Configuration: Transitioning to High-Throughput Memory Limits

Remote Write Scalability and Memory Pressure

Prometheus utilizes a queue-based mechanism to buffer time series data before transmitting it to remote storage endpoints via HTTP POST requests. In high-volume environments, the interaction between max_samples_per_send and the overall capacity within the queue_config determines the memory footprint of the remote write process.

When remote endpoint latency increases, the queue can fill rapidly. While increasing the capacity allows for larger bursts of data, it introduces a risk of significant memory pressure on the Prometheus instance, potentially impacting local scraping performance.

Given the behavior of the remote write exporter in v2.x, there is uncertainty regarding the optimal balance between queue depth and memory stability during network instability.

• What is the recommended ratio between max_samples_per_send and total queue capacity to prevent OOM events during latency spikes?
• How does the remote write mechanism prioritize sample drops when the queue reaches maximum capacity under sustained high load?