Diagnosing Envoy 503 Spikes Caused by Circuit Breaker Exhaustion

Your service mesh is throwing intermittent 503s during traffic peaks, but the upstream pods are healthy and CPU is fine. A very common cause: Envoy's circuit breakers are tripping because the default connection or request limits are too low for your actual concurrency. The fix is usually a configuration change, not more capacity — but only if you confirm the right counter is overflowing first.

One clarification up front: when an Envoy circuit breaker overflows, Envoy returns a 503 (specifically UH / UO response flags), not a 502. A 502 with UF or UC flags points at connection failure to the upstream, which is a different problem. If you're seeing 502s, check upstream health first; if you're seeing 503s with no upstream errors, read on.

Recognizing the condition

Circuit breaker exhaustion has a recognizable signature:

• 503s appear suddenly at a traffic threshold and disappear when load drops.
• Upstream services show no errors, no restarts, and low latency during the spike.
• Envoy access logs show response flag UO (upstream overflow) or URX.
• The failing requests never reach the upstream — you can confirm this because upstream request counts stay flat while 503s climb.

Symptom-to-cause table

Ordered checks

Run these against the Envoy admin interface (default port 15000 in Istio sidecars; adjust for your deployment). You need network access to the pod and permission to exec into it or port-forward.

1. Confirm the overflow counters

kubectl exec deploy/my-app -c istio-proxy -- \\
  curl -s localhost:15000/stats | grep -E 'overflow|upstream_cx_active|upstream_rq_active'

Look specifically for:

• cluster.outbound|8080||my-svc.upstream_cx_overflow — connection limit hit (max_connections).
• ...upstream_rq_pending_overflow — pending request queue full (max_pending_requests).
• ...upstream_rq_retry_overflow — retry budget exhausted.

A counter that increments in lockstep with your 503s is your culprit. If none of these move, stop — the breaker isn't your problem; go check membership_healthy and upstream logs instead.

2. Check the effective circuit breaker config

kubectl exec deploy/my-app -c istio-proxy -- \\
  curl -s 'localhost:15000/config_dump?resource=dynamic_active_clusters' | \\
  grep -A12 circuit_breakers

Envoy's historical defaults are low: max_connections: 1024, max_pending_requests: 1024, max_requests: 1024, max_retries: 3. These are per-cluster, per-host-pool in some configurations — a single busy upstream host can exhaust its share well before the aggregate looks full. Note that defaults and how control planes (Istio, Consul, Gloo) materialize these settings vary by version; verify against your actual config dump rather than assuming.

3. Correlate active vs. limit

Watch upstream_cx_active and upstream_rq_active during a load window. If active connections plateau exactly at the configured limit while 503s rise, the diagnosis is confirmed.

Fixes tied to findings

If upstream_cx_overflow is rising

Raise max_connections for the cluster. In Istio, that's a DestinationRule:

apiVersion: networking.istio.io/v1beta1\nkind: DestinationRule\nmetadata:\n  name: my-svc\nspec:\n  host: my-svc\n  trafficPolicy:\n    connectionPool:\n      tcp:\n        maxConnections: 4096\n      http:\n        http2MaxRequests: 4096\n        maxRequestsPerConnection: 100

Apply with kubectl apply -f dr.yaml, then re-check the config dump to confirm the new values propagated (usually within seconds). Risk: raising limits shifts load to the upstream. If the upstream genuinely can't handle more concurrency, you've moved the failure, not fixed it.

If upstream_rq_pending_overflow is rising

The upstream is slower than the arrival rate. Raising max_pending_requests only queues more requests and inflates tail latency — treat this as a signal to scale the upstream or reduce per-request latency, not to enlarge the queue.

If upstream_rq_retry_overflow is rising

Retries are amplifying an already-degrading upstream. Reduce retry attempts or tighten the retry budget before increasing it; a larger retry budget during an overload event causes cascading failure.

Verifying the fix

After applying changes:

1. Re-run the config dump check to confirm new limits are live.
2. Watch the overflow counter for 5 minutes under representative load: it should stop incrementing.
3. Confirm the 503 rate drops in your metrics and that upstream p99 latency did not materially increase — if it did, you traded fast failures for slow successes.

When to escalate

Escalate to the team owning the upstream service when: overflow counters keep rising after limit increases, upstream_rq_pending_overflow dominates (upstream is saturated, not the proxy), or raising limits degrades upstream latency. At that point the answer is horizontal scaling or upstream performance work, not more Envoy tuning.