In-Memory Queue vs Persistent Queue: Which Is Best for Low‑Traffic Beats?

Direct answer

For very light Beats traffic, use the persistent disk queue when losing buffered events is unacceptable. Use the in-memory queue when events are replayable or loss is tolerable. There is no published event-rate threshold at which persistent storage cost outweighs data-loss risk; at a few events per minute, storage consumption is usually negligible, so durability requirements drive the choice.

Confirmed facts and likely explanation

Confirmed facts:

• Beats provide in-memory and persistent disk queue implementations, with defaults and setting names varying by Beat and release.
• In-memory queues keep buffered events in RAM and lose them on crash or reboot.
• Persistent disk queues write events to local storage before shipping, surviving crashes at the cost of disk space and write I/O, including checkpoint activity.
• Disk-queue backpressure depends on output blocking and disk latency, not event rate alone.

Likely explanation: On a host generating only a few events per minute, a persistent queue rarely fills, so meaningful CPU or memory backpressure is unlikely. The persistent queue may still issue background disk writes even when idle, so treating it as free is unsafe without measurement.

Steps for this case

1. Decide whether buffered events can be replayed from the source. If not, select the persistent queue regardless of traffic volume.
2. Review the reference for your specific Beat and version to confirm queue.mem and queue.disk settings and the default queue type.
3. If you choose the persistent queue, configure the data path and max size on a local disk with adequate headroom.
4. Start the Beat, confirm it ships events, and inspect logs for queue or disk errors.

Example persistent queue configuration

queue.disk:
  path: /var/lib/beat-name/data
  max_size: 100MB

Backpressure on a quiet host

With minimal event generation, persistent queue backpressure is unlikely to raise host CPU or memory usage. Backpressure becomes relevant when the output is blocked or the disk is slow, causing the publisher to pause. Monitor queue size and disk write bytes rather than assuming impact from event rate alone.

Verification

Run the Beat with each queue type at your production event rate and compare host disk write bytes and Beat queue metrics. Then stop the process abruptly and compare acknowledged versus lost events for each queue type. A read-only configuration check can be run with:

filebeat test config -c /etc/filebeat/filebeat.yml

Substitute your Beat binary and config path as needed.

Missing diagnostic detail

Can the monitored source replay or regenerate events that were buffered but not shipped before the Beat stopped? If the answer is no, the persistent queue is the safer choice even for low traffic.

Note: This draft requires current verification against your Beat version and host environment before production use.