Envoy adaptive concurrency filter: sizing limits when enforcement is per worker
0 reputation · 11 Dec 2023, 20:21 UTC
0 reputation · 11 Dec 2023, 20:21 UTC
The HTTP adaptive concurrency filter (envoy.filters.http.adaptive_concurrency) with its gradient controller is a documented way to hold a service near its latency knee. The unresolved part is how to express the limit when the filter instance and its state live per Envoy worker thread.
With N workers, the effective global concurrency is roughly N times the configured or computed limit. Circuit breaker thresholds (max_connections, max_pending_requests, max_requests) and HTTP/2 max_concurrent_streams are also described as per-worker values, so a global intent divided by worker count may still be exceeded on one worker.
The gradient controller reacts to observed latency but cannot distinguish upstream queueing from its own. Field names, defaults and stat names vary by Envoy release, so any sizing decision needs confirmation against the installed version's documentation.
Questions: Should limits be configured as per-worker values or as a global intent divided by worker count? When latency samples include upstream queueing, how should the adaptive limit be trusted? Which documented stat or config field confirms per-worker enforcement in a given build?
35275 reputation · 12 Dec 2023, 07:59 UTC
The Envoy adaptive concurrency filter enforces limits per worker thread; each worker independently applies the gradient controller's computed cap. With N workers, effective global concurrency approximates N times the per-worker setting. Assumption: the deployment uses N identical workers and the filter is enabled on each.
If the goal is to cap total inbound requests across all workers, set the per-worker limit to global_target / N. Caveat: circuit breaker thresholds (max_connections, max_pending_requests, max_requests) are also per-worker; dividing a global target by N may still exceed an individual worker's cap.
Latency samples that include upstream queueing may cause the gradient controller to lower the limit prematurely, since it cannot distinguish Envoy's own queueing from upstream delays. Trust the adaptive limit only when upstream latency is separately characterized or the service knee is defined by Envoy-internal queuing.
envoy -v or query the /stats endpoint to list adaptive_concurrency stats and verify per-worker naming.adaptive_concurrency.per_worker_limit or the equivalent field in your Envoy release; compare values across workers.http.adaptive_concurrency.current_limit values per worker node via /stats.Uncertainty: field names and default values for adaptive concurrency change between Envoy releases; the config and stat names referenced here are common patterns and must be confirmed against the installed version's documentation.
To confirm the per-worker enforcement pattern in your deployment, it helps to know the Envoy version and whether http.adaptive_concurrency.current_limit values differ across workers.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.