Envoy hot restart drain stalls on long-lived connections during migration

Direct Answers

1. Does Envoy terminate the old worker after the drain timeout even when long-lived connections remain open?

Envoy sends a SIGKILL to the old worker process once the configured drain_timeout expires, regardless of whether long-lived connections such as WebSocket sessions or gRPC streams are still active. The process exit sequence prioritizes the drain timeout as a hard limit. However, the exact enforcement can vary by Envoy version and operating system; older builds may not signal the kill as consistently, and OS-level signal handling differences can affect timing.

2. Should client-side retry logic be added to absorb requests that fail while the old worker drains?

Yes. When the drain timeout expires, Envoy forcibly closes active connections, which typically results in a TCP RST or a stream-level error on the client. Adding idempotent retry logic with exponential backoff and respect for Retry-After or equivalent headers absorbs these failures without surfacing errors to end users. This is especially critical for gRPC streams and WebSocket connections, where a sudden close can lose in-flight data.

3. How do drain timeout and listener idle timeout interact during a hot restart reload?

During a hot restart the new Envoy process takes over the listener file descriptors immediately. The old process enters a draining state where it stops accepting new connections but continues to serve existing ones. The drain_timeout acts as the overall process-exit deadline, while listener_idle_timeout only closes connections that have been inactive for its configured interval. In practice the drain timeout takes precedence: if the old process has not exited within the drain timeout Envoy kills it. The listener idle timeout may close stale connections earlier but it does not override the drain timeout's hard stop.

Recommended Steps for This Case

1. Verify that your Envoy binary was built with certificate-sharing (CDS) extension support if TLS is in use; without CDS SSL state cannot be shared between old and new processes and the hot restart may fail.
2. Set the listener-level drain_timeout to a value that exceeds the longest expected long-lived connection duration. The default is 300 seconds; adjust upward for gRPC streams or WebSocket sessions that may persist for minutes to hours.
3. Configure listener_idle_timeout separately if you want Envoy to close inactive connections before the drain timeout expires. This does not affect the process-exit deadline but can reduce the number of connections the old process must drain.
4. Add client-side retry logic with idempotent operations and exponential backoff targeting 5xx or connection-reset errors that occur during the migration window.
5. Perform a test hot restart with a simulated long-lived connection (e.g. an active gRPC stream or WebSocket) and observe whether the old process terminates via SIGKILL after the specified timeout. Check Envoy logs for the drain timeout reached message.

To refine the recommendation for your specific environment it would help to know the current drain_timeout value and the Envoy version in use.