Intermittent scrape failures with no connection pool visibility

Answer to the question

Prometheus does not publish any metrics about the internal http.Client pool that it uses for scrapes and remote‑writes. The pool is governed by Go’s default http.Transport, which limits the number of idle connections per host to 2 and keeps them alive for 90 s. Because the pool is shared across all targets, a sudden spike in scrape frequency can exhaust the pool, resulting in generic errors such as connection refused or too many open files. The missing visibility is the root cause of the intermittent failures you observe.

Likely explanation

• Go’s default Transport limits idle connections per host to 2.
• Prometheus re‑uses a single http.Client for all scrapes, so high‑frequency targets can exhaust the pool.
• When the pool is exhausted, new requests are rejected or the OS limits (e.g., EMFILE) are hit.

Confirmed facts

• Prometheus constructs a global http.Client with the default Transport and exposes no configuration knobs for pool size or idle timeout.
• There are no built‑in Prometheus metrics for active, idle, or rejected connections from that pool.
• Code changes in Prometheus are out of scope; however, a custom collector can expose the needed state without breaking existing metrics.

Key metrics to expose

Implement a small Go collector that reads the Transport statistics (via expvar or internal counters) and registers them under a new namespace. The following metrics are recommended:

• prometheus_http_client_active_connections – sockets currently in use.
• prometheus_http_client_idle_connections – sockets that are idle but kept alive.
• prometheus_http_client_total_connections – sum of active and idle.
• prometheus_http_client_rejected_connections – attempts that could not acquire a socket due to pool or OS limits.

Backward‑compatible exposure

Register the collector in the same binary that runs Prometheus or ship it as a sidecar that scrapes Prometheus’ own /metrics and injects the new labels. Because the metrics live in a distinct namespace, they do not conflict with existing metrics and do not require any changes to Prometheus’ own configuration.

Suggested alerting thresholds

These thresholds give a safety margin before errors surface. Adjust them after observing your own traffic patterns.

# active connections exceed 80% of the configured idle limit
- alert: PrometheusHttpClientHighUsage
  expr: prometheus_http_client_active_connections > 0.8 * max_idle_per_host
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus HTTP client pool is heavily used"

# idle connections fall below 20% of the configured idle limit
- alert: PrometheusHttpClientLowIdle
  expr: prometheus_http_client_idle_connections < 0.2 * max_idle_per_host
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus HTTP client pool has very few idle connections"

# rejected connections spike
- alert: PrometheusHttpClientRejected
  expr: increase(prometheus_http_client_rejected_connections[5m]) > 0.05 * increase(total_scrape_attempts[5m])
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Prometheus HTTP client pool is rejecting connections"

Practical steps for your environment

1. Confirm the Prometheus version and inspect the http.Client construction in the source (or run go version and search the binary for Transport).
2. Build or obtain a custom collector that exposes the four metrics above. The collector can be compiled into the Prometheus binary or run as a sidecar that scrapes localhost:9090/metrics and forwards the new metrics.
3. Deploy the collector and verify the metrics appear under the prometheus_http_client_* namespace.
4. Simulate a high‑frequency scrape (e.g., wrk -t4 -c200 -d30s http://localhost:9090/metrics) and watch the prometheus_http_client_rejected_connections metric spike before any scrape errors show up in the logs.
5. Apply the alerting rules above and tune the max_idle_per_host constant to match the actual limit (default 2 in Go, but can be overridden in newer Go releases).
6. Monitor the alerts and adjust thresholds if you observe false positives or missed failures.

Missing diagnostic detail

To fine‑tune the thresholds, we need to know the current maximum idle connections per host configured in your Prometheus instance (typically 2, but could be different if a custom Transport is used). Please run go version and inspect the Prometheus source or binary for that value.