Configuring Logstash Persistent Queues for Production Durability

Useful answer

To protect events from being lost when Logstash crashes or the host loses power, enable the persistent queue feature by setting queue.type: persisted in logstash.yml and allocate enough local disk space for the queue.

Worked configuration and mechanism

The following snippet shows a typical production‑grade setup for a pipeline that may need to buffer tens of gigabytes of data.

# logstash.yml
queue.type: persisted
queue.max_bytes: 50gb
queue.page_capacity: 64mb
queue.checkpoint.writes: 4096
queue.drain: true

• queue.type: persisted switches the internal buffer from an in‑memory list to a disk‑based queue.
• queue.max_bytes caps the total size the queue may occupy on disk. When this limit is reached, Logstash applies back‑pressure to the input, causing the upstream source to pause or block.
• queue.page_capacity defines the size of each memory‑mapped segment (default 64 MB). Larger segments reduce the number of files but increase granularity of checkpoints.
• queue.checkpoint.writes controls how many events are accepted before an fsync forces data to stable storage. A higher value (e.g., 4096) reduces I/O overhead and can improve throughput, but widens the window of potential loss on an abrupt power failure.
• queue.drain: true tells Logstash to attempt to flush the queue during a graceful shutdown before exiting.

Logstash implements the persistent queue using memory‑mapped files (mmap). Events are written into the mapped region; the operating system’s page cache handles the actual write to disk in the background. This design lets the queue absorb short bursts with latency close to an in‑memory buffer while still providing durability.

Monitoring and practical verification

Logstash exposes queue metrics through its Monitoring API (default port 9600). To check current usage:

curl -s http://localhost:9600/_node/stats/pipelines/main/queue | \
  jq '.queue.used_bytes, .queue.free_space'

What to look for: used_bytes should rise as events arrive and drop when the output catches up. If used_bytes consistently approaches queue.max_bytes, either increase the allocated disk or improve output throughput.

A simple durability test (run in a non‑production environment) is:

1. Start Logstash with the persistent‑queue config above.
2. Feed a known number of events (e.g., 100 000) via a file input.
3. While events are still in the queue, force a crash with kill -9 <PID> (requires sufficient privileges).
4. Restart Logstash.
5. Query the output (e.g., Elasticsearch _count API) and verify that the total matches the number of events sent.

Limits and common mistakes

Storage requirements

Persistent queues rely on mmap semantics, which are not reliable on network‑mounted filesystems such as NFS or SMB. Use locally attached SSD or NVMe drives; network storage can cause file corruption, lock‑ups, or severe performance loss.

At‑least‑once delivery

The queue guarantees that an event is written to disk before being acknowledged, providing at‑least‑once delivery. If Logstash crashes after sending an event to the output but before checkpointing, the event may be resent on restart. Ensure downstream systems are idempotent (e.g., use Elasticsearch document IDs) to avoid duplicates.

Immutable queue size settings

Changing queue.max_bytes or queue.page_capacity on a live queue is not supported. To modify these values you must stop Logstash, delete the queue directory (or the specific pipeline’s subdirectory), and restart. Any events currently buffered on disk will be lost.

Pipeline isolation

When running multiple pipelines, each must have its own queue directory. Logstash creates a subdirectory under path.queue automatically, but explicitly setting path.queue: /var/lib/logstash/queue/pipeline1 for each pipeline avoids accidental sharing and potential corruption.

Windows considerations

On Windows, an unclean Logstash exit can leave lock files (*.queue) that prevent a subsequent start. Manually delete these lock files if the service fails to start after a crash.

Version compatibility

Persistent‑queue data files are not guaranteed to be compatible across major Logstash versions (e.g., 7.x → 8.x). Before upgrading, drain the queue (queue.drain: true) and verify it is empty, or plan to re‑initialize the queue after the upgrade.

Checkpoint trade‑off

A large queue.checkpoint.writes value improves throughput but increases the number of events that could be lost on a sudden power loss to the last checkpoint. Choose a value that balances your throughput needs with acceptable loss exposure.