Can Envoy prevent duplicate writes when retrying non-idempotent HTTP requests?
0 reputation · 28 Jul 2024, 07:31 UTC
0 reputation · 28 Jul 2024, 07:31 UTC
Ensure that Envoy’s retry mechanism does not cause duplicate side‑effects when retrying non‑idempotent methods such as POST, PUT, or PATCH, even if the upstream has already processed the original request before the response was lost.
Envoy currently retries based on status codes and retry_on settings without any built‑in deduplication, leaving the responsibility to the upstream or an external filter. The open design question is whether Envoy should add a native idempotency‑token filter that tracks in‑flight request hashes and suppresses duplicate retries, or keep deduplication strictly at the application layer.
What are the trade‑offs of implementing an idempotency‑token filter in Envoy versus delegating deduplication to the upstream service? How would such a filter interact with existing retry policies, retry budgets, and per‑try timeouts? Which mechanisms could be used to track in‑flight request hashes without introducing excessive memory overhead or latency?
33580 reputation · 28 Jul 2024, 15:03 UTC
No. Envoy does not natively prevent duplicate writes when it retries non-idempotent methods such as POST, PUT, or PATCH. Its retry mechanism is driven by status codes and network events, not by awareness of whether the upstream has already committed a side effect.
Confirmed: Envoy retry policies evaluate conditions like retry_on: 5xx, connection resets, or connect failures. They do not inspect request bodies, generate idempotency tokens, or maintain a deduplication table. There is no built-in filter that tracks in-flight request hashes to suppress duplicate attempts.
Likely explanation: When an upstream processes a write but the response is lost, or when a 5xx is returned after the write commits, Envoy sees only a retry-eligible failure. It will replay the identical request. The proxy cannot distinguish a request that failed before the write from one that failed after the write succeeded.
retry_on for routes handling POST, PUT, or PATCH. A route-level retry policy that omits these methods is the safest default.Idempotency-Key header). The upstream service stores the key and returns the original response for duplicate submissions. Envoy can pass the header through without modification.A custom idempotency filter would need to coordinate with Envoy's retry budget so that suppressed retries do not consume budget intended for legitimate failures. Per-try timeouts (per_try_timeout) still apply independently; if the first attempt exceeds the timeout but the upstream continues processing, a retry can race with the original request. In-flight tracking must therefore survive the interval between attempts while being cleaned up when the stream ends.
To confirm the risk, configure a test route with retry_on: 5xx and point it to a mock upstream that logs each request and returns 503 after writing. If the response is lost or a 5xx is returned, Envoy will issue a second request, producing a duplicate log entry. Review your production configuration to ensure non-idempotent methods are excluded from retry policies unless the upstream enforces idempotency.
Can your upstream service accept and enforce an idempotency key? If yes, retries can remain enabled because the application layer will absorb duplicates. If no, the recommendation is to disable retries for non-idempotent methods at the Envoy route level.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.