Nomad rolling update leaves TCP connections unresolved during migration

Question

Goal: Determine whether Nomad's default rolling update behavior, which waits for a new allocation to pass its health check before stopping the old one, ensures zero‑downtime for small applications that rely on long‑lived TCP connections.

Constraints: The update stanza can be tuned with max_parallel and health check intervals, but Nomad does not offer a built‑in connection‑draining mechanism for non‑HTTP protocols, and aggressive or misconfigured health checks may cause the old allocation to be stopped too early or the new one to be marked unhealthy, leading to a brief window where connections are dropped or duplicated.

Uncertainty: It is unclear whether the automatic promotion alone is sufficient to avoid dropped connections, or if a custom pre‑stop hook should be added to drain connections before the old allocation is terminated.

Does Nomad guarantee that the old allocation remains healthy and receives traffic until the new allocation passes its health check?
What is the impact on existing TCP connections if the old allocation is stopped before the new allocation is ready to accept them?
Should a custom pre‑stop hook be implemented to drain connections, and how would it interact with Nomad's update stanza and health‑check timing?

Wasi · Answer

Answer
Nomad’s default rolling update does not guarantee that long‑lived TCP connections stay alive during the switch; the old allocation can be stopped as soon as the new allocation passes its health check, and Nomad does not automatically drain connections for host‑network tasks.
Confirmed facts

The update stanza controls max_parallel and the health‑check interval; Nomad waits for the new allocation to be marked healthy before stopping the old one.
For host‑network allocations Nomad does not close listener sockets or wait for existing TCP connections to finish; connection draining only happens automatically when using Consul Connect bridge networking with an Envoy sidecar.
The nomad node drain command performs an explicit drain step before stopping allocations.

Likely explanation
When the new allocation passes its health check, Nomad promotes it and immediately tells the client to stop the old task. If the old task is still accepting or holding TCP connections, those connections are reset because the process receives a SIGTERM and exits, unless the application itself handles graceful shutdown.
Steps for this case

Determine whether the service runs with Consul Connect bridge networking. If yes, the Envoy sidecar will drain connections; no extra hook is needed.
If host networking is used (or Connect is not configured), add a prestop hook that either:

closes the listener socket, or
waits for existing connections to finish (e.g., by checking ss -tnp for ESTABLISHED states).

Set the update stanza to max_parallel = 1 and keep the health check interval short enough to detect readiness but long enough to avoid flapping.
After the update starts, verify drainage:

On the node hosting the old allocation run ss -tnp | grep <port> and confirm ESTABLISHED connections drop to zero before the task stops.
Check Nomad client logs for a line containing “draining allocation” or “stopping task”.

Missing diagnostic detail
Are you using Consul Connect bridge networking for this job? The answer changes whether a custom prestop hook is required.

Nomad rolling update leaves TCP connections unresolved during migration

1 answer

Answer

Confirmed facts

Likely explanation

Steps for this case

Missing diagnostic detail

Your perspective belongs here.

0 question comments