pipeline.shutdown_timeout: per‑pipeline runtime configuration missing

Question

Goal
Determine whether Logstash can expose a per‑pipeline shutdown timeout that can be adjusted while the instance is running, rather than relying on the global pipeline.shutdown_timeout setting in logstash.yml or the pipeline configuration file.
Constraints & Uncertainty
The current implementation treats pipeline.shutdown_timeout as a global value, applied uniformly to all pipelines. During a pipeline reload via the API, the shutdown logic appears to perform a full graceful shutdown followed by a restart, but the exact timing and whether the timeout is respected for each pipeline is undocumented. There is no documented per‑pipeline cancellation hook, so a slow output may cause the entire pipeline to exceed its timeout and lose events.
Questions

Is there a supported mechanism to override pipeline.shutdown_timeout for a specific pipeline at runtime?
What is the exact behavior of the shutdown timeout when a pipeline is reloaded through the API—does the timeout apply to the shutdown phase, the restart phase, or both?
Does the timeout measure from the moment the shutdown signal is received, or does it wait until all worker threads finish processing?

Tasadduq Burney · Answer

Answer
The Logstash configuration offers only a single, global pipeline.shutdown_timeout setting. There is no supported API or per‑pipeline directive that can change this timeout while the instance is running. Any attempt to reload a pipeline via the REST API will use the global timeout value for the shutdown phase.
Explanation of Shutdown Timing

When the timeout is applied: The timer starts immediately after the shutdown signal is sent to the target pipeline. It does not wait for all worker threads to finish; if the timeout expires, Logstash force‑terminates the pipeline, potentially dropping unflushed events.
Scope of the timeout: The timeout limits only the shutdown phase. The subsequent restart phase is not bound by this value and may take longer to spin up the new instance.
Effect on other pipelines: Because the setting is global, a slow output in one pipeline can cause the entire instance to terminate prematurely if the shutdown timeout is reached, even though other pipelines are still running.

Practical Implications

If you need different shutdown behavior for distinct pipelines, the recommended approach is to run separate Logstash instances, each with its own pipeline.shutdown_timeout value in logstash.yml.
Alternatively, adjust the global timeout and carefully monitor the /metrics endpoint for the pipeline_shutdown_timer_seconds metric to ensure it meets your operational window.
There are no runtime commands or API calls that can override the timeout per pipeline; you must restart the Logstash process to apply a new global value.

Next Steps
To confirm the behavior in your environment, you can set pipeline.shutdown_timeout to a low value (e.g., 1 s) in logstash.yml, restart Logstash, then trigger a pipeline reload via the REST API. Observe the shutdown duration in the logs or the pipeline_reload endpoint’s response. If the shutdown completes within the set timeout, the global setting is respected.

pipeline.shutdown_timeout: per‑pipeline runtime configuration missing

Goal

Constraints & Uncertainty

Questions

1 answer

Answer

Explanation of Shutdown Timing

Practical Implications

Next Steps

Your perspective belongs here.

0 question comments