Persistent Queues vs. Idempotent Document IDs for Duplicate Prevention

Handling At-Least-Once Delivery in Logstash

Logstash provides at-least-once delivery semantics, particularly when queue.type: persisted is enabled. While persistent queues prevent data loss during crashes by acknowledging events only after output confirmation, they introduce the risk of duplicating writes if a failure occurs after the output has processed the data but before the acknowledgement is recorded.

The Trade-off

To mitigate these duplicates, developers must choose between relying on the default behavior of the output plugin or implementing deterministic identification via the fingerprint filter and the document_id setting in the Elasticsearch output.

The decision depends on the availability of stable unique keys within the source data. Without a reliable unique field, the fingerprint filter may generate inconsistent IDs, while relying solely on persistent queues accepts the risk of duplicates during retry cycles or bulk request failures (e.g., 429 responses).

• Persistent Queues: Ensures durability but may replay events already written downstream.
• Deterministic IDs: Ensures idempotency by overwriting existing documents, but requires a stable source key.

Under what conditions is it preferable to accept duplicates and deduplicate downstream rather than implementing fingerprint-based IDs within the pipeline? Does the overhead of calculating fingerprints for every event outweigh the cost of downstream deduplication in high-throughput environments?