Envoy circuit breaker: intermittent upstream connection pool exhaustion under aggregate limits

Identifying the Primary Binding Constraint

The primary driver of overflow is identified by which specific Envoy counter is incrementing during the exhaustion event. Because Envoy applies limits at the cluster level rather than the endpoint level, the pattern of overflow reveals the bottleneck:

• upstream_cx_overflow: The max_connections limit is the driver. This occurs when the total number of active TCP connections across all hosts in the cluster exceeds the threshold.
• upstream_rq_pending_overflow: The max_pending_requests limit is the driver. This typically indicates that connections are being established, but the upstream is too slow to accept them, filling the queue.
• HTTP/2 Multiplexing Issues: If upstream_cx_active remains low but you see request-level failures, the bottleneck is likely the HTTP/2 max_concurrent_streams setting or max_requests_per_connection. The latter forces connection churn, which can cause intermittent spikes in upstream_cx_overflow as Envoy rapidly opens and closes connections.

Per-Priority Overrides vs. Aggregate Restructuring

Whether to use per-priority overrides depends on your traffic criticality. Aggregate limits are shared; if low-priority background traffic spikes, it can starve high-priority user requests.

Use per-priority overrides if: You have a clear distinction between traffic classes (e.g., high for user-facing API calls, low for asynchronous telemetry) and need to guarantee headroom for the former regardless of the latter's volume.

Restructure aggregate limits if: The exhaustion is caused by a general mismatch between Envoy's capacity and the upstream's ability to handle concurrent load. Simply increasing limits without scaling the backend may shift the failure from Envoy to the upstream TCP backlog.

Isolation Methodology for Load Testing

To isolate aggregate-limit exhaustion from per-host or upstream-imposed limits, use a controlled reduction of the host pool:

1. Single-Host Isolation: Reduce the cluster to a single healthy backend host. This removes the variable of load-balancing distribution and confirms if the aggregate limit is hit based on total volume alone.
2. Saturation Ramp: Gradually increase request concurrency while monitoring the ratio of upstream_cx_active to max_connections.
3. Churn Simulation: Set max_requests_per_connection to a very low value to simulate high connection turnover and verify if this triggers upstream_cx_overflow despite low concurrent request volume.

Diagnostic Requirement

To refine this recommendation, please provide the current value of max_requests_per_connection and whether you are using HTTP/1.1 or HTTP/2 for upstream communication.