Persistent queue event ordering constraints during rolling node migrations

Ordering Guarantees in Persistent Queues

No, Logstash does not guarantee strict FIFO (First-In, First-Out) ordering when multiple worker threads are configured to process events from a single persistent queue. While the persistent queue ensures that events are written to disk in the order they are received, the subsequent processing phase is concurrent.

The Concurrency Gap

When pipeline.workers is set to a value greater than 1, multiple threads pull events from the queue simultaneously. Because different events may require varying amounts of processing time—or may be delayed by network jitter during output transmission—events can arrive at the destination out of sequence. This is a fundamental trade-off between throughput and ordering.

Configuration Constraints

There is currently no configuration setting to enforce strict sequential processing while maintaining a multi-threaded execution model. To guarantee that events reach the output plugin in the exact order they were queued, you must restrict the pipeline to a single worker:

pipeline.workers: 1

During a rolling migration, this restriction ensures ordering but creates a processing bottleneck, which may lead to the persistent queue filling up more quickly if the incoming data rate exceeds the single-threaded processing capacity.

Verification and Risk Mitigation

To verify the impact of concurrency on your specific data set, you can run a scoped test using a sequenced generator:

• Test Setup: Use a generator input producing incrementing integers.
• Output: Write to a local file.
• Analysis: Compare the output file sequence against the expected linear sequence while toggling pipeline.workers between 1 and higher values.

Assumption: This behavior applies to standard Logstash persistent queue implementations (typically found in versions 7.x and 8.x). If you are using a distributed broker (like Kafka) in front of Logstash, ordering is managed at the partition level, not the Logstash queue level.

To provide a more tailored recommendation for your migration window, please specify if your output plugin supports idempotent writes or if the destination system can re-order events based on a timestamp.