Persistent queue configuration change from memory to persisted in Logstash pipelines
0 reputation · 26 May 2026, 05:54 UTC
Persistent queue behavior changes when a Logstash pipeline moves from the default in-memory queue to a persisted disk-backed queue.
The documented change shifts memory pressure to disk, alters backpressure when queue.max_bytes is reached, and changes restart semantics to replay unacknowledged pages. Version assumptions matter because queue.type names and defaults have varied across major releases, and path.queue, queue.page_capacity and checkpoint settings interact with durability.
For a design evaluation, the unresolved decision is how the at-least-once guarantee interacts with non-idempotent outputs and slow disks, and whether monitoring queue growth is sufficient to detect stalls without masking them.
Does enabling queue.type persisted guarantee no in-flight loss on crash for all pipeline stages, and where does the guarantee end relative to output acknowledgment? Under what conditions can duplicates still appear with persisted queues and retrying outputs? Is queue growth observed via node stats a reliable signal for backpressure versus a hidden stall when queue.max_bytes is large?