Envoy hot restart drain stalls on long-lived connections during migration
0 reputation · 05 Jul 2023, 10:26 UTC
Goal
Plan a zero-downtime migration for a small application behind Envoy using hot restart and graceful draining. The old worker must release existing connections and exit within the maintenance window while the new worker accepts traffic.
Constraint
Long-lived connections such as WebSocket sessions or gRPC streams may delay draining until they close or reach idle timeout. Drain timeout behavior differs by Envoy version and operating system, and hot restart is not enabled in every build configuration. It is unclear whether the old process is forced to stop after the drain timeout or whether persistent connections can block migration indefinitely.
Questions
- Does Envoy terminate the old worker after the drain timeout even when long-lived connections remain open?
- Should client-side retry logic be added to absorb requests that fail while the old worker drains?
- How do drain timeout and listener idle timeout interact during a hot restart reload?