Logstash Persistent Queue: Architecture Note for Durable In-Flight Buffering

The Problem: Event Loss During Restarts

In a standard Logstash configuration, the queue is memory-based. If the Logstash process crashes or the server restarts, any events that have been accepted from an input but not yet delivered to the output are permanently lost. For pipelines ingesting critical data via Beats, TCP, or file inputs, this creates a reliability gap.

The takeaway: Enabling the Persistent Queue (PQ) moves the buffer from volatile RAM to local disk, ensuring that events survive restarts and providing a mechanism to handle downstream back-pressure without dropping data.

Requirements

• Durability: In-flight events must persist across process restarts and system reboots.
• Back-pressure Handling: The system must signal upstream producers to slow down when the buffer is full.
• Zero-Loss Guarantee: Once an event is acknowledged by the input plugin, it must be delivered to the output.
• Minimal Infrastructure: The solution must operate without requiring an external message broker (like Kafka or RabbitMQ).

Smallest Suitable Design

The most efficient way to meet these requirements is to enable the built-in persisted queue in the Logstash configuration. This design relies on a local directory to store queue pages.

# logstash.yml
queue.type: persisted
queue.path: /var/lib/logstash/queue
# Default page size is 64MB; usually sufficient for most workloads
# queue.page_capacity: 64mb
# 0 indicates the queue is limited only by available disk space
# queue.max_events: 0

In this design, Logstash writes events to disk in pages. This creates a durable buffer between the input stage and the filter/output stages of the pipeline.

Trust and Data Boundaries

The persistent queue introduces a new data boundary on the local filesystem. The queue files are owned by the Logstash OS user and should be restricted via filesystem permissions to prevent unauthorized access.

Security Warning: Logstash does not provide built-in encryption for the persistent queue. If the events contain sensitive data (PII or secrets), the underlying storage must be encrypted using OS-level tools such as LUKS (Linux) or BitLocker (Windows). Once data enters the queue, it is treated as trusted for downstream processing, but the input source remains untrusted.

Operational Checks

To ensure the queue is functioning and not approaching a failure state, monitor the Logstash HTTP metrics API (typically http://<host>:9600/_node/stats?pretty) using the following indicators:

• persistent_queue.page_data.bytes: Tracks the total size of data currently queued.
• persistent_queue.events: Tracks the number of individual events held in the queue.

Health Thresholds:

• Disk Space: Maintain free disk space on the queue partition that is at least 2x the observed page_data.bytes to allow for page rotation and compaction.
• Stall Detection: Alert if persistent_queue.events remains static for more than 30 seconds while inputs are active; this indicates the pipeline is blocked or disk I/O is saturated.

Failure Modes and Redesign Triggers

Common Failure Modes

• Disk Exhaustion: If the partition hosting queue.path fills up, Logstash will block new writes. This triggers back-pressure, causing upstream inputs (like Beats) to stop sending data. The system resumes automatically once space is freed.
• Queue Corruption: In rare cases of hard crashes or filesystem errors, queue files may become corrupted. Logstash will fail to start. The only recovery is to wipe the queue directory (rm -rf /var/lib/logstash/queue/*), which results in the loss of all buffered events.

When to Redesign

The local persistent queue is a "middle-ground" solution. You should migrate to an external broker (e.g., Apache Kafka) if you encounter these conditions:

• I/O Bottlenecks: Sustained disk I/O latency exceeding 10ms that slows down the overall pipeline throughput.
• Capacity Issues: Queue depths regularly exceeding 80% of available disk space.
• High Availability Needs: A requirement for multi-node durability; a local PQ is tied to a single host and cannot be accessed by other Logstash nodes.

Practical Verification

To verify the implementation (assuming Logstash 8.x), perform the following steps on a test instance with sudo permissions:

1. Configure a simple pipeline: input { beats { port => 5044 } } output { stdout { codec => rubydebug } }.
2. Apply the queue.type: persisted settings in logstash.yml and start the service: sudo systemctl start logstash.
3. Send a burst of events using Filebeat.
4. Query the metrics API to confirm persistent_queue.events is increasing.
5. Stop the service: sudo systemctl stop logstash.
6. Restart the service: sudo systemctl start logstash.
7. Verify that the events sent before the stop command are now appearing in the stdout logs.

Rollback

Reverting to a memory queue is a state-changing operation that deletes buffered data.

1. Stop the Logstash service.
2. Remove the persisted data: sudo rm -rf /var/lib/logstash/queue.
3. Update logstash.yml to queue.type: memory.
4. Restart Logstash.