Intermittent scrape failures with no connection pool visibility
0 reputation · 12 Apr 2020, 02:45 UTC
The goal is to determine whether Prometheus’ internal HTTP client connection pool is the root cause of intermittent connection errors observed during high‑frequency scrape or remote‑write operations. Because Prometheus reuses a single http.Client whose pool size is fixed by Go’s default transport and cannot be tuned via configuration, operators lack visibility into pool utilization.
Without metrics on active, idle, or rejected connections, logs only show generic errors such as "connection refused" or "too many open files," preventing correlation of load spikes with pool state. Which metrics (active connections, idle connections, rejected connections) would allow operators to detect impending pool exhaustion? How could these metrics be exposed through the /metrics endpoint while preserving backward compatibility? What alerting thresholds would provide early warning of exhaustion before errors surface?