Persistent Queues vs. Idempotent Document IDs for Duplicate Prevention

Short answer

Accept duplicates and deduplicate downstream when (a) your events have no stable unique key, (b) the expected duplicate rate is low, and (c) your downstream store either tolerates duplicates or has a cheaper dedup path than a per-event fingerprint. Fingerprint-based document_id is the right choice when you have a deterministic key and need exactly-once semantics at the index level — but it is not free, and it is impossible without a stable key.

Confirmed facts vs. likely explanation

Confirmed: with queue.type: persisted, Logstash acknowledges events only after output confirmation, so a crash or a failed bulk request (for example, HTTP 429s from Elasticsearch) can cause events that were already written to be replayed. Persistent queues give you durability, not exactly-once delivery. Setting document_id in the Elasticsearch output turns indexing into an idempotent upsert on _id, collapsing replays into rewrites of the same document.

Likely, but workload-dependent: the fingerprint filter's cost is usually small relative to JSON parsing and bulk-request I/O, so in many pipelines it is not the bottleneck. The claim that fingerprinting is too expensive should be treated as a hypothesis to measure, not a default assumption.

When downstream dedup wins

• No stable source key. If events lack a unique, immutable field (no event UUID, no natural key), a fingerprint of the payload can collide for legitimately identical-but-distinct events, silently merging real data. Accepting duplicates is then the only safe option.
• Low duplicate rate. Replays typically happen only around crashes and backpressure events. If duplicates are a small fraction of volume and your analytics tolerate them (or a scheduled dedup job cleans them), the operational simplicity of queues-only beats maintaining fingerprint config.
• Batch-tolerant consumers. If a downstream stage (Spark, a warehouse load, a streaming aggregation with a dedup window) already processes data in bulk, deduplicating there amortizes the cost across batches instead of paying per-event CPU in the hot path.
• High-throughput, latency-sensitive pipelines. Per-event hashing adds CPU on every event, forever, to protect against a rare failure. If profiling shows the filter measurably reduces worker throughput at your target event rate, moving dedup off the critical path is reasonable.

When fingerprint IDs win

• A stable key exists (transaction ID, event UUID, immutable natural key).
• Duplicates cause real harm: billing events, counts that must be exact, or downstream systems without idempotent writes.
• You want the guarantee enforced at write time rather than trusting a cleanup job.

Verify before deciding

1. Measure the filter's cost: compare pipeline throughput and CPU with and without the fingerprint filter using Logstash's node/pipeline metrics APIs (GET _node/stats/pipelines).
2. Measure your actual duplicate rate: run with persistent queues, force a restart under load, and count duplicate documents in Elasticsearch (e.g., aggregate on a payload hash after the fact).
3. Estimate downstream dedup cost at your volume, and confirm it keeps up with peak ingest — a dedup stage that falls behind erases the benefit.

One caveat: the Elasticsearch output's idempotency depends on document_id being set; with auto-generated IDs, every replay creates a new document. If you skip fingerprinting, confirm your actual consumers can absorb duplicates, because Elasticsearch itself will not.