Diagnosing Upstream Connection Pool Exhaustion and Circuit Breaker Triggers in Envoy

Recognizable condition

When Envoy logs circuit_breaker events and the upstream 5xx error rate exceeds the per‑cluster max_requests_threshold, upstream connections are observed in the TRIP_OPEN state. This condition means the circuit breaker has tripped because the connection pool cannot satisfy the incoming request volume. The TRIP_OPEN state indicates that further connections are rejected until the breaker resets, which can cause increased latency and a cascade of additional 5xx responses.

Diagnostic table

Observed symptom	Relevant config field	Possible cause
Frequent connection failures	max_connections	Pool size limit reached
High latency spikes	max_pending_requests	Requests queued beyond allowed pending count
Sudden 5xx surge	retry_params	Retry logic causing extra connections
Circuit breaker log entry TRIP_OPEN	connection_breaker	Breaker triggered by any of the above

Ordered checks

1. Verify runtime circuit breaker overrides via /runtime endpoint. Execute curl localhost:9901/runtime and inspect the tree for entries such as envoy.filters.network.EnvoyListenerConfiguration. If a runtime key is set to a value higher than the static configuration, it may raise max_connections or max_pending_requests unintentionally, causing the pool to exceed its intended limit. Multiple overrides can stack, so verify that no contradictory values exist.
2. Confirm per‑cluster circuit breaker config in static or xDS config. Open the bootstrap file or the xDS resource that defines the upstream cluster. Verify that the max_connections, max_pending_requests, and the connection_breaker block are set to values that match your capacity planning. Remember that recent Envoy releases treat per‑cluster and per‑route breakers differently; ensure you are looking at the correct resource and that migration from per‑cluster to per‑route has been accounted for.
3. Check upstream health status and outlier detection. Use curl localhost:9901/stats to view counters like upstream_cx_active and upstream_cx_closed. A rising upstream_cx_closed count may indicate health‑checking failures that shrink the effective pool, while a low upstream_cx_active suggests the pool is under‑utilized. Also examine the outlier detection thresholds (consecutive_5xx, consecutive_timeout) for misconfigurations.
4. Validate listener and filter chain for overrides. Ensure that any listener‑level or filter‑chain‑level circuit breaker configuration does not conflict with the cluster‑level settings. A per‑route breaker that is more aggressive than the cluster breaker can cause unexpected tripping during traffic spikes, especially when the route configuration changes dynamically.

Fixes tied to findings

• If runtime overrides are too permissive, disable or adjust them through the runtime API: curl -X POST localhost:9901/runtime/override -d 'envoy.filters.network.EnvoyListenerConfiguration.max_connections: 15000'. After updating, send a reload signal to force Envoy to re‑read the configuration. This step is essential when a previous operator set a higher limit for testing and forgot to revert it.
• Adjust static or xDS cluster settings directly. Edit the cluster definition to increase max_connections or max_pending_requests, or to remove the connection_breaker block entirely if it is not required for your workload. After editing, perform a config reload (e.g., SIGHUP or admin /reload) to apply the change without downtime.
• Reset stale runtime values using the admin API: curl -X POST localhost:9901/admin/runtime_modify -d 'envoy.filters.network.EnvoyListenerConfiguration.max_connections=0'. Setting the value to zero forces Envoy to discard any runtime override and fall back to the static configuration, which can clear accidental overrides that keep the pool artificially small.
• Align retry parameters with circuit breaker limits. Reduce max_retries or increase retry_backoff so that the number of in‑flight requests does not exceed max_pending_requests. For example, set max_pending_requests to 20 when the default retry count would otherwise create 100 pending requests, thereby preventing the pool from filling too quickly.

Escalation criteria

When the 5xx error rate remains above the circuit breaker threshold after the above adjustments, proceed with the following escalation steps:

1. Analyze upstream service latency. High response times can cause the pool to fill faster than expected, leading to premature tripping. Use upstream service metrics to identify slow endpoints.
2. Review network path and load‑balancer metrics for packet loss, TCP retransmissions, or DNS resolution delays that limit the number of simultaneous connections. Tools such as tcpdump or cloud provider network health dashboards can reveal hidden bottlenecks.
3. Check for per‑route circuit breaker migration in recent Envoy releases. Per‑route configurations may behave differently from per‑cluster settings, especially after an upgrade, and can cause unexpected tripping. Review the release notes for changes to how per‑route breakers are evaluated.
4. Engage the upstream service team to tune their own connection limits or to investigate application‑level request bursts that generate sudden spikes in in‑flight requests. Sometimes the root cause lies in the application rather than Envoy.

Verification steps

After any change, verify that the circuit breaker state transitions as intended:

# 1. Capture baseline counters
curl localhost:9901/stats | grep circuit_breaker

# 2. Apply a configuration change (example: raise max_connections)
curl -X POST localhost:9901/admin/runtime_modify -d 'envoy.filters.network.EnvoyListenerConfiguration.max_connections=20000'

# 3. Reload runtime to make the change effective
curl -X POST localhost:9901/runtime_modify -d 'envoy.filters.network.EnvoyListenerConfiguration.max_connections=20000'

# 4. Run a controlled load test
hey -n 10000 -c 200 http://localhost:8080/

# 5. Verify counters have not crossed the trigger threshold
curl localhost:9901/stats | grep circuit_breaker

Watch the circuit_breaker.open counter; it should move to closed after the test if the pool is adequately sized. If the counter stays high, revisit the earlier checks and consider the escalation steps.