Filebeat and Elasticsearch: Event Duplication Risks During Network Retries

At-Least-Once Delivery and Acknowledgment

Filebeat utilizes an at-least-once delivery model to ensure data integrity. By default, the agent waits for an acknowledgment from Elasticsearch before removing events from its internal queue. When a bulk request fails, the agent employs exponential backoff and splits the bulk to retry only the failed items.

Pipeline Acknowledgment Trade-offs

There is a design consideration regarding the pipeline.ack feature. Shifting the acknowledgment responsibility to the Beats pipeline can reduce latency but may increase the risk of duplicate indexing if the agent shuts down abruptly before the pipeline clears.

Configuration Constraints

• Memory pressure increases when pipeline.ack is enabled, as events remain in memory longer.
• The interaction between flush.min_events and network failures may lead to events being flushed before the acknowledgment is received.

Given these behaviors in Beats 7.x and later, how does the system resolve the conflict between minimizing latency via pipeline acknowledgments and preventing duplicate writes during a crash? Under what specific conditions does the bulk_max_size configuration influence the likelihood of duplicates upon restart?