Preserving Event Order in Logstash: When to Use a Single Worker

Why Order Matters

When Logstash runs with the default setting of pipeline.workers equal to the number of CPU cores, each worker thread pulls events from the input queue and processes them in parallel. Parallelism boosts throughput but it also means the order in which events exit the pipeline is no longer the same as the order they entered. For downstream consumers such as time‑series databases, log aggregation services, or any system that relies on a strict chronological sequence, this scrambling is unacceptable.

What Controls Order in Logstash?

Logstash does not provide a dedicated pipeline.ordered flag. The only built‑in way to guarantee that events leave the pipeline in the same order they arrive is to run the pipeline with a single worker thread: pipeline.workers: 1. When only one worker processes the event stream, the order is preserved by construction.

Setting pipeline.workers to a value greater than one introduces parallelism, which allows events to be processed out of order. There is no official setting that forces ordering while still using multiple workers.

Concrete Example: Single vs Multi‑Worker

Below is a minimal pipeline that tags each event with a sequential counter and outputs the result to stdout. By running the same pipeline with different worker counts we can observe the effect on ordering.

# test.conf
input {
  generator {
    lines => ["event1", "event2", "event3", "event4", "event5"]
    count => 5
    pace => 0
  }
}

filter {
  mutate { add_field => { "seq" => "0" } }
  ruby {
    code => "event.set('seq', event.get('seq').to_i + 1)"
  }
}

output {
  stdout { codec => line }
}

Run the pipeline with a single worker:

logstash -f test.conf -b 1 -w 1

The output will be:

1
2
3
4
5

Now run the same pipeline with four workers:

logstash -f test.conf -b 1 -w 4

Because events are processed in parallel, the order of the numbers may appear shuffled, e.g.:

3
1
5
2
4

Trade‑offs and Limitations

• Throughput impact: A single worker serializes all processing. For high‑volume inputs this can become a bottleneck and may violate performance SLAs.
• Resource usage: With pipeline.workers: 1 you are not utilizing all available CPU cores, which can be wasteful in multi‑core environments.
• Version considerations: The behavior described applies to Logstash 8.x and later. Earlier releases (pre‑8.0) also use pipeline.workers in the same way; there is no pipeline.ordered flag in any official release.
• Batch size interaction: The -b flag sets the batch size. Even with a large batch, a single worker still preserves order because it processes the entire batch sequentially.

Verifying Order Preservation

1. Start the pipeline with -w 1 and record the output sequence.
2. Restart with -w 4 and compare the output. Any deviation from the original sequence indicates that ordering is not preserved.
3. Use logstash -V to confirm you are running a supported Logstash version (8.x or later). If you need to preserve order on a multi‑worker pipeline, you must accept the performance trade‑off or implement external ordering logic.

Practical Takeaway

If your downstream system requires strict ordering, set pipeline.workers: 1 in the pipeline configuration or override it on the command line with -w 1. For workloads where throughput is more important than order, keep the default or a higher worker count and allow events to be processed out of order. Always test in a staging environment that mirrors production traffic and monitor input and output rates to ensure your chosen configuration meets performance goals.