Question
Route timeout behavior during retry attempts in Envoy
Zo Echo
0 reputation · 05 Feb 2023, 12:53 UTC
59.9K views1
Loading…
0 reputation · 05 Feb 2023, 12:53 UTC
Envoy uses the timeout field within the Route configuration to define the maximum duration the proxy waits for an upstream response. When retries are enabled, the interaction between the overall route timeout and the per_try_timeout configuration determines when a request is finally terminated with a 504 Gateway Timeout.
There is uncertainty regarding the precise calculation of the remaining time budget for subsequent retry attempts when a per-try timeout is shorter than the total route timeout. Specifically, it is unclear if a per-try timeout reset extends the total request lifetime or if the global route timeout acts as a hard ceiling regardless of the retry count.
The total route timeout acts as a hard ceiling for the entire request lifecycle. It includes the time spent on all previous failed retry attempts, as well as the time spent on the current attempt. The timer starts when the first downstream request is received and is not reset when a retry is triggered.
Envoy evaluates two timers concurrently during a request: the global route timeout and the per_try_timeout. The request is terminated as soon as either timer expires.
A request is terminated by the global route timeout before the final per_try_timeout expires when the remaining time budget is less than the configured per-try limit. For example, if the route timeout is 2 seconds and two previous attempts have already consumed 1.7 seconds, the third attempt will be terminated after 0.3 seconds, even if the per_try_timeout is set to 500ms.
To verify this behavior in a test environment (assuming Envoy v1.18+), use the following configuration logic:
envoytimeout: 2s and per_try_timeout: 500ms.per_try_timeout (at 500ms each). The third attempt will be terminated by the route timeout after approximately 1 second of total elapsed time (or the remaining budget), resulting in a 504 Gateway Timeout.You can distinguish between these two timeout sources using Envoy statistics:
| Statistic Counter | Meaning |
|---|---|
retry_timeout |
The individual attempt exceeded the per_try_timeout. |
downstream_cx_total_timeout |
The overall request exceeded the route-level timeout. |
Note: Please specify if you are using a version of Envoy prior to 1.18, as older versions may handle per-attempt limits differently.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 05 Feb 2023, 15:06 UTC
To practically differentiate between a global route timeout and a per-try timeout during debugging, you should monitor the Envoy access logs for specific response flags. While both may result in a 504 Gateway Timeout, the internal trigger differs.
In standard Envoy configurations (v1.18+), look for the following indicators in the access log:
per_try_timeout was reached for the current attempt. If retries are remaining and the global budget allows, Envoy will trigger another attempt.A critical configuration pitfall to avoid is setting the per_try_timeout equal to or greater than the global timeout. Doing so effectively nullifies the retry policy for timeout-related failures, as the global ceiling will always be hit before a single attempt can fail and trigger a retry.