Does an upstream timeout trigger Envoy retries when timeout is missing from retry-on?

Question

I'm configuring an HTTP route retry policy in Envoy (assuming 1.27+ syntax) and want predictable behavior when an upstream service responds too slowly. My route sets a request timeout and a retry policy with num_retries and a retry_on list containing 5xx and gateway-error.

The uncertainty: it's unclear whether an upstream request that exceeds the configured timeout is treated as a retryable failure by default, or whether timeout must be explicitly added to retry_on. I also see references to connect-failure as a separate condition, which makes it ambiguous whether a slow response and a failed connection establishment are handled differently by the same policy.

A second concern is the interaction with downstream timeouts: if the downstream client timeout fires before the upstream timeout, does Envoy abort the request entirely and skip any remaining retry budget?

Specifically:

Is timeout excluded from the default retry_on set, so no retry occurs on a slow upstream unless it is listed explicitly?
Does each retry attempt reset the per-try timeout while still counting against num_retries?
Which stats counters (e.g., upstream_rq_retry) confirm a retry was actually attempted after a timeout?

Tasadduq Burney · Answer

Direct answerNo. With Envoy 1.27+ HTTP route retry policy, an upstream request that exceeds the configured timeout is not retried unless timeout is explicitly listed in retry_on. Your current list of 5xx and gateway-error will retry on those response codes only. A slow upstream that times out is treated as a terminal failure for that attempt.Confirmed factsretry_on is a whitelist. Omitting timeout means a request that hits the per-try timeout is not eligible for retry.connect-failure is a separate condition from timeout. Connect-failure covers inability to establish TCP/TLS, timeout covers no response within the allocated window.upstream_rq_retry increments whenever Envoy initiates a retry, regardless of trigger. upstream_rq_timeout increments when a per-try timeout occurs.If the downstream client closes the connection or a downstream timeout fires, Envoy aborts the active request and cancels any pending upstream retries. The remaining retry budget is not used.Likely explanation and practical behaviorEach retry attempt resets the per-try timeout timer. The total elapsed time is still bounded by the overall request timeout configured on the route. If the overall timeout is shorter than per-try timeout × (num_retries + 1), retries can be cut short by the outer deadline.Retrying on timeout can cause retry storms when the upstream is overloaded and slow. If you do want timeout retries, add timeout to retry_on and ensure the overall request timeout is longer than the sum of per-try attempts.Steps needed for this caseCheck the route config. If you want timeout retries, change retry_on to include timeout, e.g. retry_on: [5xx, gateway-error, timeout].Verify timeout scope. Confirm whether your timeout is the route-level request timeout or a per-try timeout. That changes how the retry budget is consumed.Validate with stats. With retry_on: [5xx] only, force an upstream hang and confirm upstream_rq_timeout increments but upstream_rq_retry does not. Add timeout to retry_on and confirm upstream_rq_retry increments on the same hang.One missing diagnostic detail that changes the recommendation: is the timeout you are referring to the route request timeout or a per-try timeout set via retry_policy.retry_on_per_try_timeout / per-try settings? The interaction with num_retries differs.

Does an upstream timeout trigger Envoy retries when timeout is missing from retry-on?

1 answer

Direct answer

Confirmed facts

Likely explanation and practical behavior

Steps needed for this case

Your perspective belongs here.

0 question comments