Prometheus 2.44 Remote_write: Unexpected Connection Pool Exhaustion Despite Queue Config Adjustments

Goal

Determine why a Prometheus 2.44 deployment intermittently exhausts the HTTP client connection pool used for remote_write even after tuning queue_config parameters.

Constraints & Uncertainty

• The queue_config block (max_samples_per_send, max_shards, batch_send_timeout) directly influences how many samples are queued before a send, but it does not expose the number of open connections.
• Active connections are only visible via debug logs; no built‑in metric tracks pool pressure.
• Defaults for queue_config vary between minor releases, potentially masking or worsening exhaustion.

Open Questions

1. Does increasing max_samples_per_send reliably reduce idle connections, or can it lead to larger bursts that still exhaust the pool?
2. Would exposing a dedicated metric such as prom_remote_write_active_connections help operators detect and mitigate pool pressure?
3. What idle_timeout configuration would provide a balance between connection reuse and preventing stale connections without triggering exhaustion?