Stop Cascading Failures with Envoy Circuit Breakers

The problem: one slow service takes down everything

Your downstream API starts responding slowly. Clients keep retrying. Envoy keeps opening new connections and queuing requests. Soon every worker thread is blocked waiting for that one service, and healthy services sharing the same proxy instance starve for resources. This is a classic cascading failure.

The useful takeaway: Envoy can stop this automatically. Circuit breaking lets you set hard ceilings per upstream cluster—maximum concurrent requests, pending requests, and connections. When a ceiling is hit, Envoy fails fast instead of piling more load on an already struggling service.

How Envoy circuit breaking works

A circuit breaker tracks the current load to an upstream cluster. Envoy maintains counters for active requests, pending (queued) requests, and connections. Each counter has a configurable limit. When a limit is reached, the circuit is considered open for that priority, and new requests are rejected immediately—typically with a 503 Service Unavailable—rather than being forwarded.

Key thresholds, set per cluster in the circuit_breakers field:

• max_requests: maximum concurrent active requests.
• max_pending_requests: maximum requests waiting in the queue.
• max_connections: maximum concurrent connections to the upstream.
• max_retries: maximum retry attempts for a single request.

Circuit breaking is stateful: once a threshold is crossed, Envoy tracks the open state until load drops below the limit. It works alongside retries, not instead of them. If a circuit is open, Envoy suppresses retries and fails fast, which protects the upstream but means your application must handle 503s gracefully.

Worked configuration example

This example assumes Envoy running with the v3 API (recent stable releases). Field names and defaults changed across versions, so check your release notes before copying.

Add a circuit_breakers block inside a cluster definition:

name: product-api
type: STRICT_DNS
connect_timeout: 5s
load_assignment:
  cluster_name: product-api
  endpoints:
  - lb_endpoints:
    - endpoint:
        address:
          socket_address:
            address: 10.0.0.42
            port_value: 8080
circuit_breakers:
  thresholds:
  - priority: DEFAULT
    max_requests: 200
    max_pending_requests: 50
    max_connections: 300
    max_retries: 3

Where to run: Edit the Envoy configuration file on the host or container running Envoy. You need write access to the config directory and permission to reload or restart the Envoy process.

Expected checks: After applying the config and sending traffic, query Envoy’s admin stats endpoint. Replace <ADMIN_PORT> with your admin port (commonly 9901):

curl -s http://localhost:<ADMIN_PORT>/stats | grep product-api.circuit_breaker

Look for counters such as state_open, active_requests, and pending_requests. Exact stat names include the cluster name and priority, so verify the prefix in your Envoy version.

Risk: If thresholds are set below normal peak concurrency, Envoy returns 503s for legitimate traffic. Start with values above your observed p99 concurrency and tighten gradually.

Diagnosing an open circuit

When the circuit opens, Envoy stops forwarding new requests to that cluster. You can confirm this from metrics and logs:

• circuit_breaker.state_open flips to 1 while the breaker is open.
• circuit_breaker.active_requests sits at the limit and does not grow.
• Envoy access logs show 503 responses with the upstream cluster marked as overloaded.

Retry interaction matters here. If you also configure retries, an open circuit suppresses retry attempts, so clients see a single 503 instead of a delayed storm of retries. Tune your retry budget separately so retries do not quietly consume the same circuit-breaker headroom.

Practical check: Generate load above max_requests in a staging environment, then watch /stats/prometheus. When the breaker opens, state_open should rise and new requests should receive 503s immediately. When load drops, the circuit should close and normal routing resumes.

Trade-offs and limitations

Circuit breaking is not a substitute for health checks. Health checks remove unhealthy endpoints from rotation; circuit breakers limit total load. Use both.

Because the breaker is stateful, a temporary spike can flip the circuit open and keep it there briefly even after the spike ends. Thresholds that are too aggressive cause unnecessary latency and failed requests under normal load.

Configuration schemas have changed across Envoy releases. A circuit_breakers block that works in one version may be rejected or ignored in another. Validate against the documentation for your running binary.

Finally, circuit breaking masks symptoms. Frequent openings are a signal that your upstream is undersized or your application has a dependency problem, not that the proxy is misconfigured.

What to do next

1. Instrument first: export Envoy circuit-breaker stats to Prometheus and alert on state_open duration.
2. Set initial thresholds from observed production concurrency, not guesses.
3. Test in staging: exceed max_requests and verify 503s, suppressed retries, and automatic recovery.
4. Document rollback: keep the previous config file and reload Envoy with it if thresholds cause user-facing errors.

Circuit breaking will not fix a slow service, but it can prevent one slow service from taking down your entire platform.