Persistent queue configuration change from memory to persisted in Logstash pipelines

Answer to the design questions

No. Enabling queue.type: persisted does not guarantee zero in-flight loss for all pipeline stages, and the durability guarantee ends at queue acknowledgment, not at output acknowledgment.

Duplicates can still appear with persisted queues when outputs are retrying or non-idempotent. Queue growth from node stats alone is not a reliable sole signal for backpressure versus a hidden stall when queue.max_bytes is large.

Confirmed facts

• Persistent queue makes queued pages durable on disk and replays unacknowledged pages after a restart.
• The queue marks a page complete only after the worker finishes the pipeline and the checkpoint advances. Durability therefore stops at the queue boundary before the output stage.
• Persistent queue provides at-least-once delivery. The design is intentionally at-least-once, not exactly-once.
• Node stats expose queue.events_in, queue.events_out, current size and max_bytes. These metrics reflect queue state, not downstream output health.

Likely explanation

Events already dequeued to workers and held in filters or outputs are not covered by the queue. A crash after dequeue but before worker acknowledgment can lead to reprocessing. Output plugins may succeed, partially succeed, or fail without feedback to the queue, so in-flight output state is not protected.

Duplicates occur when Logstash restarts after an output has succeeded but before the queue page is checkpointed, when a worker crashes mid-processing, or when a non-idempotent output is retried due to transient errors. Slow disk checkpoint latency increases the reprocessing window.

With a large queue.max_bytes the queue can absorb bursts for an extended period, masking downstream stalls. Growth can reflect normal buffering, while disk I/O saturation can cause a hidden stall with stable or slowly growing queue metrics.

Version and configuration assumptions

Queue type names, defaults and checkpoint behavior have varied across major Logstash releases. Durability depends on queue.type, path.queue permissions and free space, queue.max_bytes, queue.page_capacity and checkpoint settings. Persistent queue shifts pressure to disk I/O; slow disks increase latency and can create hidden backpressure not visible in memory metrics.

Steps needed for this case

1. Inspect pipeline configuration for queue.type, path.queue, queue.max_bytes, queue.page_capacity and checkpoint settings.
2. Query node stats for queue events in, out, current size and compare growth against max_bytes over time, alongside disk I/O and output latency.
3. Simulate a controlled restart with test events to observe replay behavior and whether events are reprocessed.
4. Review output plugin retry, error handling and idempotency characteristics for the specific outputs in use. Plan for idempotent writes or deduplication if non-idempotent.

One missing diagnostic detail that changes the recommendation: Logstash major version and the specific output plugins in use. Version determines queue defaults and checkpoint semantics, and output type determines whether at-least-once delivery will cause visible duplicate side effects.