Logstash Persistent Queues: Durability Between Input and Output

The problem: memory queues disappear on crash

Logstash pipelines accept events at an input, transform them in filters, and hand them to outputs. By default the buffer between input and filter/output lives in JVM memory. If the process dies unexpectedly, anything still in that queue is gone. Persistent Queues (PQ) move that buffer to disk so a crash or restart does not discard events Logstash already accepted.

The useful takeaway: PQ is a durability feature, not a throughput feature. It protects against process loss, not against disk hardware failure, and it adds disk I/O cost.

Requirements: when memory is not enough

Two conditions usually justify moving from the default in-memory queue to PQ:

• Volume exceeds available JVM heap. When input rate is higher than output capacity, events accumulate. An unbounded memory queue can drive heap pressure and out-of-memory errors.
• Zero-data-loss requirement. If the downstream system cannot tolerate gaps after a Logstash restart, accepted events must survive on disk.

If neither condition applies, the simpler in-memory queue may remain the better default.

The smallest suitable design

The minimal change is to set the queue type and give Logstash a dedicated directory. In logstash.yml:

queue.type: persisted
path.queue: /var/lib/logstash/queue

Create the directory before starting Logstash. Run these commands on the Logstash host as root or with sudo:

sudo mkdir -p /var/lib/logstash/queue
sudo chown logstash:logstash /var/lib/logstash/queue
sudo chmod 750 /var/lib/logstash/queue

Expected check: after Logstash starts, the directory exists and contains queue files. Risk: if ownership or permissions are wrong, Logstash may fail to start.

Logstash persists events to the queue files before filters and outputs consume them. This behaves like a write-ahead log: the event is on disk first, then processed.

Trust and data boundaries

Keep the queue directory on a separate volume from the Logstash binaries and logs. That separation gives three practical boundaries:

• Capacity isolation. Log or binary growth cannot starve the queue of disk space.
• Independent scaling. You can resize the queue volume without touching the application install.
• Access control. Restrict the directory to the Logstash service account so other local processes cannot tamper with queued events.

PQ does not protect against disk hardware failure. If the underlying disk or array fails without RAID or external replication, queued events are lost. Durability requirements above process crashes need storage-level redundancy.

Operational checks

Queue depth and drain rate

Watch the queue metric in Logstash pipeline stats. A queue depth that keeps rising and never drains means output throughput is lower than input ingestion. That is a bottleneck signal, not a PQ failure.

Crash survival test

Simulate a hard kill to confirm events survive. Find the main PID on the Logstash host:

systemctl show --property=MainPID --value logstash

Then, as root or with sudo, send an uncatchable kill signal using the PID from the previous command:

sudo kill -9 <PID>

Restart Logstash and verify that events accepted before the kill are processed afterward. Risk: this test is abrupt and should use disposable data in a non-production environment.

Disk behavior under load

During a controlled burst, confirm the queue directory grows and then drains as outputs catch up. Sustained growth without drain indicates the output stage cannot keep pace.

Failure modes and conditions that change the design

Failure mode	Effect	Condition that changes the design
Disk exhaustion	Pipeline stalls or Logstash crashes when the queue volume fills	If backlog cannot be bounded, add capacity, increase output throughput, or apply backpressure upstream
Throughput overhead	Maximum sustainable throughput is lower than memory queues because every event hits disk	If occasional loss is acceptable and raw throughput is the priority, stay with the in-memory queue
Permission errors	Logstash fails to start when it cannot write to the queue directory	Fix ownership before restart; automate directory creation in provisioning
Hardware failure	Queued events are lost despite PQ	Add RAID, replication, or an external durable buffer if the requirement is stronger than process-crash survival

How to verify the result

After enabling PQ and restarting, compare the number of events accepted at the input with the number delivered to the output over a known interval. Then repeat the hard-kill test. If post-restart delivery matches the pre-kill input count and the queue depth returns to zero, the configuration is working for process-crash durability.

Version note

This note assumes a recent Logstash 7.x or 8.x release. Configuration keys, default paths, and monitoring API endpoints can vary by version. Verify the exact settings against the documentation for your installed version before production rollout.