Enabling Logstash Persistent Queues: A Practical Guide for Data Durability

The Problem: In‑Memory Volatility

Logstash’s default queue is held in RAM. If the process crashes, the host loses power, or a restart occurs during a surge, any events still in that buffer are lost. For audit trails, security logs, or any use case where data loss is unacceptable, this default behaviour is a hard failure point.

The Solution: Persistent Queues (PQ)

Persistent Queues replace the in‑memory buffer with a disk‑based segment store. Every event is written to disk before the filter and output stages consume it. On restart, Logstash reads the last checkpoint and continues processing from where it left off, preserving data integrity.

Implementation Steps

1. Edit the Logstash Global Configuration

Locate logstash.yml (typically /etc/logstash/logstash.yml on Linux). You need root or sudo privileges to modify it.

# logstash.yml
queue.type: persisted          
queue.max_bytes: 4gb           
queue.checkpoint_interval: 5000

What each line does:

• queue.type: persisted turns on PQ for all pipelines on the node.
• queue.max_bytes limits the total disk space the queue can consume. If the limit is reached, Logstash will apply backpressure to input plugins.
• queue.checkpoint_interval controls how often the queue writes a checkpoint to disk. The default is 5 seconds.

2. Verify the Queue Directory

On most installations, persistent queue files are stored in /var/lib/logstash/queue. After a Logstash restart, this directory should contain segment files (e.g., queue-1-0-0-0.bin) and a checkpoint file.

# List queue files (run as the Logstash user or with sudo)
ls -R /var/lib/logstash/queue

If you see segment files, PQ is active.

Testing Durability

To confirm that PQ survives a hard crash:

1. Start Logstash with PQ enabled.
2. Send a high‑volume test stream (e.g., via logstash-input-file or tcp).
3. While events are still being queued, kill the process forcefully: kill -9 $(pgrep -f logstash).
4. Restart Logstash.
5. Check the output (e.g., Elasticsearch) for the missing events. If they reappear, PQ worked.

Look for log entries such as Using persistent queue on startup to confirm the feature is in use.

Performance Trade‑offs

Disk I/O Latency

Writing to disk is slower than RAM. The ingestion rate is limited by the underlying storage. SSDs or NVMe drives mitigate this. If you observe sustained high latency, consider upgrading the storage medium.

Backpressure and Stalls

When queue.max_bytes is reached, Logstash stops accepting new events until the output drains space. This backpressure can slow the source system. Monitor disk usage with df -h or similar tools.

Checkpoint Overhead

Frequent checkpoints (small queue.checkpoint_interval) increase disk writes. The default 5 s interval is a balance between durability and performance.

Plugin Compatibility

Most current Logstash plugins are compatible with PQ, but some legacy inputs that rely on synchronous in‑memory handoffs may not. Verify each plugin’s documentation or test in a staging environment before production. If a plugin fails, you may need to disable PQ for that pipeline or upgrade the plugin.

Comparison: Memory vs. Persistent Queue

Feature	Memory Queue (Default)	Persistent Queue (PQ)
Durability	Lost on crash	Preserved on disk
Latency	Very low (RAM)	Higher (disk I/O)
Resource Risk	Out‑of‑Memory errors	Disk‑space exhaustion
Backpressure	None	Applied when full
Recovery	Starts fresh	Resumes from checkpoint

Rollback Procedure

If PQ introduces unacceptable latency or disk pressure, revert to the memory queue:

1. Edit logstash.yml and change queue.type back to memory.
2. Restart Logstash.
3. Important: Any unprocessed events that remain on disk will be orphaned. Ensure the queue has drained or back up the segment files before changing the setting if those events are critical.

Common Mistakes to Avoid

• Setting queue.max_bytes too low for your ingest volume – the queue will fill quickly and cause stalls.
• Assuming PQ works on very old Logstash releases (< 7.4) – the feature was introduced in 7.4 and later.
• Using a slow or fragmented disk – disk I/O becomes the choke point.
• Overlooking plugin compatibility – some legacy inputs may not honour the persistent queue.