Preventing Logstash Data Loss with Persistent Queues

The Risk of the In-Memory Buffer

By default, Logstash uses an in-memory queue to buffer events between the input and output stages. While this provides high throughput, it creates a critical vulnerability: if the Logstash process crashes or the server restarts, every event currently sitting in that memory buffer is permanently lost. Similarly, if your downstream destination—such as Elasticsearch—experiences a prolonged outage, the memory buffer can fill up, leading to backpressure that may cause your source systems to drop logs.

The solution for production environments where data durability is non-negotiable is the Persistent Queue (PQ). This feature shifts the buffering mechanism from volatile RAM to a disk-based write-ahead log, ensuring that events survive process restarts and downstream downtime.

How Persistent Queues Change Data Flow

When you enable PQ, Logstash no longer holds the primary event buffer in memory. Instead, as soon as an event is received by the input plugin, it is written to a dedicated directory on the local disk. Only after the event is safely persisted is it passed to the filter and output stages.

This architecture provides two primary advantages:

• Durability: If the Logstash service is killed or the host reboots, the events remain on disk and are processed immediately upon restart.
• Burst Capacity: Disk space is significantly cheaper and more abundant than RAM. You can configure a PQ to hold gigabytes of data, allowing Logstash to absorb massive spikes in log volume without crashing or rejecting incoming traffic.

Implementation and Configuration

Persistent Queues are configured globally in the logstash.yml file. Because this changes how the core engine handles memory and disk I/O, it cannot be toggled within an individual pipeline configuration file.

Configuration Example

To enable PQ, modify your logstash.yml (typically located in /etc/logstash/ on Linux) with the following settings:

# Enable the persistent queue mechanism
queue.type: persisted

# Define the maximum size of the queue on disk
# Ensure the underlying mount point has sufficient space
queue.max_bytes: 4gb

# Optional: Define the path for the queue data
# path.queue: /var/lib/logstash/queue

Execution Requirements:

• Permissions: You must have root or sudo privileges to edit logstash.yml and restart the Logstash service.
• Restart: Changes to logstash.yml require a full restart of the Logstash process to take effect.
• Risk: If queue.max_bytes is set higher than the available physical disk space, the operating system may experience I/O errors, potentially locking the filesystem.

Verifying the Queue State

Once enabled, you should verify that Logstash is actually utilizing the disk and monitor the queue depth during downstream outages. You can check the status via the Logstash Monitoring API. Run the following command from the local terminal (assuming the API is enabled on port 9600):

curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'

Look for the logstash.queue.persisted.queue.size metric. In a healthy state with no backlog, this number should be near zero. During a downstream outage, you will see this number climb as events accumulate on disk.

The Trade-off: Throughput vs. Durability

Moving from memory to disk introduces a performance penalty. Every event must now be written to the disk (I/O) before it can be processed. In high-volume environments, the disk I/O becomes the primary bottleneck.

Feature	Memory Queue (Default)	Persistent Queue (PQ)
Latency	Ultra-low	Higher (Disk I/O overhead)
Crash Recovery	Data lost	Data preserved
Buffer Size	Limited by RAM	Limited by Disk Space

If your environment requires millions of events per second and you have a separate queuing layer (like Apache Kafka) upstream, the memory queue may be preferable. However, if Logstash is your first point of ingestion, the durability of PQ is usually the correct engineering choice.

Actionable Summary

To move to a durable Logstash architecture: set queue.type: persisted, allocate a queue.max_bytes value that fits your disk capacity, and monitor the _node/stats API to ensure your disk isn't filling up during destination outages. If you need to revert to memory queues, change the type back to memory and restart the service; note that any remaining data in the persistent queue will be ignored.