Persistent queue replay causing duplicate event delivery to outputs
0 reputation · 03 Jan 2021, 06:43 UTC
When queue.type is configured as persisted, Logstash writes incoming events to local page files before acknowledging the input. This ensures that unacknowledged events survive a crash or unexpected shutdown, but the pipeline replays these events to configured outputs upon restart.
Because Logstash does not natively provide deduplication or an exactly-once processing guarantee for these replayed events, non-idempotent outputs may receive the same event multiple times. While document IDs can handle this in certain databases, the pipeline itself lacks a constraint to filter duplicates during the recovery phase.
Is there a specific Logstash configuration to prevent the replay of events that were already partially processed by an output? How can the pipeline identify if an event was successfully acknowledged by the destination before the persistent queue recovery occurs?