Prometheus 2.44 Remote_write: Unexpected Connection Pool Exhaustion Despite Queue Config Adjustments

Direct answer

queue_config does not limit or expose the remote_write HTTP connection pool. It controls sample batching and shard concurrency only. Pool exhaustion in Prometheus 2.44 is most likely caused by many concurrent shards opening keep-alive connections to slow or unresponsive remote endpoints, with retries opening connections faster than idle ones are reclaimed.

Likely explanation versus confirmed facts

Likely explanation

• Each remote_write target runs one sender per shard, controlled by max_shards. More shards means more concurrent HTTP connections.
• Slow remote endpoints or TLS handshakes hold connections open. Retries and batch_send_timeout can open new connections before old ones close.
• queue_config has no idle_timeout and does not set keep-alive reclamation; that depends on the Prometheus build's HTTP transport defaults.
• Raising max_samples_per_send reduces request frequency but can increase per-connection hold time and burst size, so it is not a reliable pool fix.

Confirmed facts for Prometheus 2.44

• max_samples_per_send, max_shards, and batch_send_timeout influence queuing and sender concurrency; none sets an open-connection limit.
• Prometheus exposes remote_storage metrics for queue length, shard count, sent and failed samples, and send duration, but no built-in active-connections or pool-pressure metric.
• Active connections are visible only through debug logs or external socket or proxy metrics.

Steps for this case

1. Reduce sender concurrency. Temporarily lower max_shards in the remote_write config, or reduce the number of remote_write targets. Validate with promtool check config, then reload Prometheus.
2. Bound slow sends. Set send_timeout so Prometheus abandons stalled sends instead of holding connections indefinitely.
3. Size for bursts. Align capacity and batch_send_timeout with observed remote endpoint throughput, not just local sample volume.
4. Raise max_samples_per_send only after confirming the remote endpoint can absorb larger batches without increased latency.

Verification

Run these read-only checks during or after an exhaustion event:

• Count established connections to each remote endpoint: ss -tan state established | awk '{print $5}' | cut -d: -f1 | sort | uniq -c
• Check queue pressure using remote_storage metrics such as queue length, shard max, failed samples, and send duration.
• Enable --log.level=debug briefly and correlate connection open or reuse messages with send failures and retries, then return to info.

Review note

This is a conservative draft. Verify exact remote_storage metric names and HTTP transport defaults in your Prometheus 2.44 build before applying changes to production.

Missing diagnostic detail

Provide the active connections per remote_write endpoint at exhaustion and the dominant failure mode: timeout, connection reset, or connection refused. That detail determines whether the fix is lowering max_shards, raising send_timeout, or scaling the remote endpoint itself.