Choosing Between Logstash In‑Memory and Persistent Queues for At‑Least‑Once Delivery

Decision and Constraints

When designing a Logstash pipeline you must decide whether to keep events only in memory or to write them to disk using a persistent queue. The choice affects latency, throughput, durability and storage requirements.

Supported Options

Option	queue.type	Typical Latency	Throughput	Durability	Disk Impact
In‑memory queue	memory	Low (sub‑millisecond)	Highest, limited only by CPU and network	None – events lost on abrupt stop	None
Persistent queue	persisted	Higher (disk write latency)	Lower, bounded by I/O speed of queue directory	At‑least‑once – events survive crashes	Writes page files; requires sufficient space and reliable storage

Trade‑offs

• Latency vs durability: Persistent queues add the time to write each page to disk, which can increase end‑to‑end latency by a few milliseconds on fast SSDs and more on slower drives.
• Throughput vs I/O: Sustained high event rates may saturate the disk if queue.page_capacity is small or checkpoint frequency is high.
• Storage planning: You must provision path.queue for at least queue.max_bytes plus overhead for page files and checkpoints.

Concrete Implementation

Enable a persistent queue by editing the pipeline settings (usually in logstash.yml or a pipeline‑specific .conf file).

1. Choose a fast, reliable volume for the queue, e.g. an SSD mounted at /var/log/logstash/queue. Ensure the Logstash process user can read and write there.
2. Add or modify the following settings:

queue.type: persisted
path.queue: /var/log/logstash/queue
queue.page_capacity: 65536   # default, can be tuned
queue.max_bytes: 1gb         # adjust to expected burst size
queue.checkpoint.writes: 1024 # default, trade latency for durability

Place the snippet in /etc/logstash/logstash.yml (global) or inside a pipeline definition if using pipeline.id‑specific files.

Restart Logstash to apply the change:

sudo systemctl restart logstash

Verify that the queue activated by checking the Logstash logs:
sudo journalctl -u logstash | grep -i \"persistent queue\"

You should see lines similar to:
[INFO ] logstash.agent - Starting persistent queue
[INFO ] logstash.agent - Loaded checkpoint from /var/log/logstash/queue/checkpoint

Validation Steps
To confirm at‑least‑once behavior:

Start Logstash with a simple test pipeline:

input { stdin { } }
output { stdout { codec => rubydebug } }

Run it in the background, send a burst of events (e.g., using yes \"test event\" | head -n 10000 piped to Logstash’s stdin), then kill the process abruptly:
kill -9 $(pgrep -f logstash)

Restart Logstash with the same configuration.
Observe the output: the same number of events should appear after restart, indicating that the persisted events were replayed.
Check for duplicate lines; at‑least‑once delivery may produce duplicates if the pipeline acknowledged events after the checkpoint.

Limitations and Practical Checks

Disk pressure: If the queue directory fills, Logstash will stall and stop accepting new events. Monitor free space with df -h /var/log/logstash/queue and set an alert at 80 % usage.
I/O latency: On network‑attached storage or slow HDDs, persistent queues can become a bottleneck. Use iostat -x 1 to watch await times; if they exceed a few milliseconds consider tuning queue.page_capacity (larger pages reduce write frequency) or moving to faster media.
Checkpoint frequency: Lowering queue.checkpoint.writes makes checkpoints more frequent, increasing durability but also I/O. Adjust based on your acceptable replay window.

Rollback
If you decide persistent queues are not suitable, you can revert to the in‑memory queue:

Stop Logstash.
Remove or rename the queue directory to prevent accidental reuse:

sudo mv /var/log/logstash/queue /var/log/logstash/queue.disabled


Edit logstash.yml and set queue.type: memory (or delete the persisted‑queue lines).
Start Logstash again.

Any events that were still on disk will be lost after the directory is removed or renamed, which is the expected trade‑off when moving back to a non‑durable queue.