ReadMeFeed / Community knowledge

#persistent-queue

Real questions. Useful conversations. Find the people who know your stack.

Question1 answer3,395 followingPosted 03 Jan 2021, 06:43 UTC147.1K views4,843 votes

Persistent queue replay causing duplicate event delivery to outputs

When queue.type is configured as persisted , Logstash writes incoming events to local page files before acknowledging the input. This ensures that unacknowledged events survive a crash or unexpected shutdown, but the pipeline replays these events to configured outputs upon restart. Because Logstash does not natively provide deduplication or an exactly-once p

Community & Careers · Editorial question

Community & Careers

Question1 answer1 comment3,606 followingPosted 28 Sept 2020, 13:12 UTC123.2K views4,332 votes

owner answered

Persistent queue recovery verification limits in OpenTelemetry Collector

The OpenTelemetry Collector can buffer spans, metrics, and logs to disk using the file_storage extension and a persistent queue so that telemetry survives a backend outage and drains after restart. The queue provides at-least-once delivery, which means an unclean shutdown may cause the Collector to resend events already accepted by the backend. The Collector

Community & Careers · Editorial question

opentelemetry opentelemetry-collector file-storage