Envoy circuit breaker: intermittent upstream connection pool exhaustion under aggregate limits

Diagnosing intermittent upstream connection pool exhaustion

Envoy bounds upstream connection pressure through per-cluster, per-priority circuit-breaker limits such as max_connections, max_pending_requests, max_requests, and max_retries. When a limit is reached, Envoy rejects or queues work instead of opening more connections. Intermittent exhaustion appears in cluster-level counters: upstream_cx_overflow (connection limit hit), upstream_rq_pending_overflow (pending-request limit hit), and upstream_rq_retry_overflow, alongside gauges like upstream_cx_active and upstream_rq_pending_active.

Constraints and ambiguity

• Thresholds are aggregated per cluster and priority, not per endpoint. One hot endpoint can consume the entire connection budget while others idle, producing bursty, hard-to-reproduce exhaustion.
• HTTP/2 and HTTP/3 multiplexing interact with max_requests_per_connection and the HTTP/2 max_concurrent_streams setting, forcing connection churn that may masquerade as a connection-count problem.
• Outlier detection ejection reduces the healthy-host pool under the same aggregate cap, and panic-mode routing can shift traffic priorities unexpectedly.
• Counters reveal which limit fired but not whether raising it or reducing churn is the correct fix. Per-priority tracking also means default-priority traffic can exhaust while high-priority retry traffic retains headroom; adding per-priority overrides is a policy choice, not a documented default.

Which binding constraint—connection count (max_connections) or concurrent stream/request capacity (max_pending_requests, max_requests_per_connection, HTTP/2 max concurrent streams)—is the primary driver of the observed overflow pattern? Does the per-priority exhaustion behavior warrant explicit per-priority circuit-breaker overrides, or should the aggregate limits be restructured? What controlled load-test methodology best isolates aggregate-limit exhaustion from per-host or upstream-imposed limits?