Forced-open circuit breaker state after runtime max_connections adjustment

Question

The investigation addresses whether Envoy's circuit breaker can transition from forced-open state when the listener's connection limit is adjusted at runtime, without requiring a full configuration reload. This behavior impacts traffic routing when downstream retry mechanisms are active, especially in deployments that combine listener-level and per-route limits.
Version-specific implementations may evaluate the threshold differently, and the interaction between retry_policy settings and the forced-open condition is not fully documented. Additionally, runtime updates to connection limits might be subject to caching or admin-triggered reloads, creating uncertainty about when the state reflects the new value.
Does the circuit breaker re-evaluate the updated limit on each new connection attempt, or does it maintain the prior state until an admin reload? How does the downstream retry_policy influence the threshold during concurrent connection teardowns? What is the minimal change required to reset the circuit breaker state without disrupting established connections?

Tasadduq Burney · Answer

AnswerThe forced‑open flag stays set after you change the listener’s max_connections at runtime; it is not automatically cleared by the new limit.Confirmed factsEnvoy treats the forced‑open flag as an override that is evaluated before normal threshold checks.Runtime updates to max_connections modify the normal threshold but do not touch the forced‑open flag.The flag persists until an explicit admin reset (or process restart).retry_policy settings affect only the pending‑request counter (max_pending_requests) and never influence the forced‑open flag.Steps to reset without dropping existing connectionsIdentify the listener or cluster name whose circuit breaker is forced‑open (e.g., my_listener).Issue an admin reset command:curl -X POST "http://localhost:9901/circuit_breakers?reset=my_listener"Verify the flag cleared:curl -s "http://localhost:9901/circuit_breakers?format=json" | jq '.[] | select(.name=="my_listener")'Existing connections remain alive during the reset; only new connection attempts are re‑evaluated against the updated max_connections limit.Missing diagnostic detailIf you are using a per‑route circuit breaker instead of a listener‑level one, please confirm the exact name (route‑specific cluster) so the reset command targets the correct breaker.

Forced-open circuit breaker state after runtime max_connections adjustment

1 answer

Answer

Confirmed facts

Steps to reset without dropping existing connections

Missing diagnostic detail

Your perspective belongs here.

0 question comments