In-Memory Queue vs Persistent Queue: Which Is Best for Low‑Traffic Beats?

Goal

Determine the most cost‑effective queue strategy for Elastic Beats when the monitored traffic is minimal.

Constraints

Beats provide two documented queue implementations:

• In‑memory queue – low CPU and disk I/O, but loses buffered events on crash.
• Persistent disk queue – writes events to local storage before shipping, adding durability at the expense of higher I/O and storage cost.

The decision hinges on whether the operational overhead and storage consumption of the persistent queue justify its durability benefits for very light traffic.

Unresolved Decision

For workloads generating only a few events per minute, the documentation does not specify a traffic threshold that clearly favors one queue type over the other. Additionally, the impact of queue backpressure on upstream producers remains uncertain.

Questions

1. At what event‑rate does the persistent queue’s storage cost outweigh the risk of data loss with an in‑memory queue for low‑traffic Beats?

2. How does the persistent queue’s backpressure behavior affect the CPU and memory usage of a host with minimal event generation?