Persistent queue replay causing duplicate event delivery to outputs

Direct Answer

No, Logstash has no configuration option to prevent replay of events that were already sent to an output but not yet acknowledged by the pipeline. The persistent queue tracks acknowledgment at the pipeline level—after all filters and outputs succeed—not per output. If Logstash crashes after an output accepts an event but before the pipeline checkpoints, that event will be replayed on restart.

Why This Happens

With queue.type: persisted, Logstash provides at-least-once delivery. Events are written to page files on disk and acknowledged only when the entire pipeline (filters + all outputs) completes successfully. The queue does not record which individual outputs succeeded. A crash between output write and pipeline checkpoint leaves the event unacknowledged, so recovery replays it.

Practical Solution: Make Outputs Idempotent

Since you cannot stop the replay, you must ensure that replayed events do not create duplicates at the destination. This is done per output:

• Elasticsearch: Set document_id to a deterministic value (e.g., %{[@metadata][fingerprint]}).
• JDBC / databases: Use a primary key or unique constraint on a stable event field.
• Kafka: Enable enable.idempotence=true and set a transactional.id on the output; use the event key for partitioning.
• HTTP / custom outputs: Include a deduplication key in the payload and handle it at the receiver.

Generating Deterministic IDs

Add a fingerprint filter early in the pipeline to create a stable hash from one or more event fields:

filter {
  fingerprint {
    source => ["message", "host", "@timestamp"]
    target => "[@metadata][fingerprint]"
    method => "SHA256"
    key => "optional-secret-for-hmac"
  }
}

Then reference %{[@metadata][fingerprint]} as the document_id (Elasticsearch) or primary key (JDBC). The fingerprint adds CPU overhead; benchmark at your event volume.

What Doesn't Help

• Dead Letter Queue (DLQ): Captures events that fail output processing, not events that succeeded at the output but missed the pipeline checkpoint.
• Disabling persistent queues (queue.type: memory): Eliminates replay duplicates but loses durability during crashes.
• External deduplication (Redis, etc.): Possible but adds latency and operational complexity; only justified for strict exactly-once requirements.

One Diagnostic Detail

Which output plugins are you using? The idempotency strategy differs per plugin (Elasticsearch document_id, JDBC primary key, Kafka idempotent producer, etc.), so knowing your outputs lets me confirm the exact configuration needed.