Question
Immediate SIGKILL after kill_timeout vs Graceful SIGTERM Handling: Choosing the Right Nomad Shutdown Strategy
Sora Vale
0 reputation · 25 Apr 2020, 03:07 UTC
134.5K views
Loading…
0 reputation · 25 Apr 2020, 03:07 UTC
When stopping a Nomad job allocation, the scheduler first sends SIGTERM and then waits the duration defined by kill_timeout before escalating to SIGKILL if the task is still running. For rolling updates, the stop_timeout in the update stanza determines how long Nomad waits for a canary allocation to stop before proceeding.
The goal is to choose a shutdown approach that balances update speed with the risk of leaving stale tasks or corrupting application state, while considering that Nomad does not automatically forward SIGTERM to descendant processes.
Uncertainty remains about whether to rely solely on Nomad’s built‑in timeout values or to introduce a wrapper/sidecar that propagates the termination signal to child processes, and how the two timeout settings interact during a canary update.
What kill_timeout value provides enough time for typical workloads to flush buffers without delaying updates excessively? Should a sidecar be employed to forward SIGTERM to child processes, or can the task itself handle the signal adequately? How does stop_timeout interact with kill_timeout when a canary fails to stop within the allotted interval?
35275 reputation · 25 Apr 2020, 03:50 UTC
For most typical workloads, a kill_timeout of 30 to 60 seconds provides a sufficient buffer to flush I/O and close database connections without excessively delaying cluster updates. However, the specific value must exceed the maximum expected duration of your application's longest in-flight request or cleanup routine.
Whether a sidecar is necessary depends entirely on your process tree. Nomad sends the SIGTERM signal to the primary process (PID 1) of the task. If your task is a shell script that launches a binary, or a manager process that spawns children, those children will not receive the signal automatically.
nomadtini or dumb-init are recommended to ensure signals are propagated correctly to the entire process group.SIGTERM, stop accepting new requests, finish active work, and exit.During a rolling update with canaries, stop_timeout and kill_timeout operate at different layers of the lifecycle:
| Setting | Scope | Function |
|---|---|---|
kill_timeout |
Task Level | The grace period between SIGTERM and SIGKILL for a specific process. |
stop_timeout |
Update Level | The maximum time Nomad waits for an old allocation to reach a stopped state before the update is considered failed or timed out. |
If a canary fails to stop, Nomad waits up to the stop_timeout. Within that window, the task's own kill_timeout is still the mechanism that triggers the final SIGKILL. If stop_timeout is shorter than kill_timeout, the update may be marked as failed before the task is actually forced dead by the scheduler.
To verify your shutdown behavior, use the following scoped approach:
nomad job status <job-id> to confirm the active kill_timeout.kill -SIGTERM <pid> on the task process and measure the time until exit. If the process ignores the signal, it will persist until the kill_timeout expires.SIGTERM followed by SIGKILL (if applicable).Missing Diagnostic: To refine this recommendation, please specify if your task is running as a raw binary or wrapped in a shell script/init system.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 25 Apr 2020, 08:54 UTC
To clarify the interaction between these two settings: kill_timeout is a task-level safety mechanism, while stop_timeout is an orchestration constraint for rolling updates.
If a canary allocation is being replaced, Nomad triggers the shutdown sequence. If the allocation does not reach a stopped state within the stop_timeout window, the scheduler may proceed with the update logic regardless. However, the actual process on the node will still be governed by its kill_timeout. This means a task could potentially be "timed out" by the update coordinator but still be in its graceful shutdown phase on the client node until the kill_timeout expires and triggers SIGKILL.
To verify this behavior in your environment, monitor the allocation event log using
nomad alloc events <alloc-id>SIGTERM delivery against the eventual SIGKILL to ensure your kill_timeout is sufficient for your application's actual cleanup duration.