Logstash Persistent Queues: Keeping Events Safe When Traffic Spikes

The spike that empties your dashboard

You deploy a new service, traffic triples overnight, and by morning your Elasticsearch index is missing a chunk of events. Logstash did not throw an error; it simply fell behind. By default, Logstash buffers events in memory between the input stage and the filter/output workers. That buffer is fast, but it is volatile. When a traffic spike outpaces the output plugin—say, Elasticsearch is indexing slowly or briefly unavailable—the in-memory queue fills. Logstash then applies back-pressure by blocking inputs, which may cause upstream senders to buffer or drop. Worse, if the Logstash process crashes or restarts while that memory queue is full, every event sitting in RAM is gone.

Persistent queues move the durability line

Persistent queues (PQs) change where Logstash draws the line between “accepted” and “safe.” With queue.type: persisted, incoming events are written to disk before the pipeline processes them. Logstash stores them in a series of page files under the directory you specify. A write-ahead log records the state so that, after a crash, Logstash can resume from the last durable checkpoint rather than starting from zero.

The architecture decouples ingestion from processing. Input plugins keep accepting events as long as there is disk space. A separate drain thread reads from the page files and feeds batches to the pipeline workers. If your output stalls, the queue grows on disk instead of blocking the input immediately. When the output recovers, the drain thread catches up.

A concrete configuration

Enable PQs in logstash.yml (typically /etc/logstash/logstash.yml). You need write access to the queue directory and permission to restart the Logstash service.

queue.type: persisted
queue.path: "/var/lib/logstash/queue"
queue.page_capacity: 64mb
queue.max_events: 0

• queue.type: persisted switches from the in-memory queue to disk-backed pages.
• queue.path must point to a fast, reliable disk. An SSD or NVMe volume is strongly preferred; a slow network mount can become the bottleneck.
• queue.page_capacity sets the size of each page file. The default is 64mb; larger pages mean fewer files but coarser allocation.
• queue.max_events: 0 removes the event-count ceiling, allowing the queue to grow until disk space runs out.

After editing the file, restart Logstash. On a systemd host, run sudo systemctl restart logstash. If you run Logstash manually, stop the process and start it again with the same configuration. The queue directory is created on first start if it does not exist.

Verifying the behavior

Configuration is only half the job; you need to confirm that events survive a stall and a restart.

1. Baseline the API. Logstash exposes a monitoring API on port 9600 by default. Run curl -s "http://localhost:9600/_node/stats/pipeline?pretty" and note the current queue.events value.
2. Create a burst. Use a load tool such as ab, wrk, or a custom script to push events through an input (for example, HTTP or Beats) faster than the output can index them.
3. Stall the output. Stop the downstream service—for example, sudo systemctl stop elasticsearch—or block its network port. Do not stop Logstash.
4. Confirm accumulation. Re-run the stats API call. You should see queue.events climb. Also check the queue directory: du -sh /var/lib/logstash/queue should show growing page files.
5. Test crash safety. While the output is still stalled, restart Logstash (sudo systemctl restart logstash). When it comes back, the queue should resume from the last checkpoint, not from zero.
6. Resume and drain. Restart the output. Watch queue.events fall back toward the baseline. Then compare event counts at the source and destination to ensure nothing was lost.

Trade-offs and limitations

Persistent queues are not free. Every event incurs a disk write and a later disk read. On adequate hardware this adds single-digit to low tens of milliseconds of latency; on saturated disks it can add much more. If your use case is real-time alerting where 50 ms matters, measure before committing.

Storage is the second constraint. An unbounded queue (queue.max_events: 0) will happily consume all available disk. If the output is down for hours, the queue can fill the volume. When that happens, Logstash blocks inputs, and the back-pressure moves upstream to your senders. A bounded queue avoids this but trades disk exhaustion for event drops once the cap is reached.

Finally, PQs protect against process crashes and output stalls, not against disk failure. If the queue volume itself fails, events are lost. For higher durability, replicate the queue directory at the storage layer or ship to a secondary Logstash instance.

What to do next

If your pipelines see bursty traffic or occasional output outages, enable persistent queues on a dedicated SSD volume. Set a bounded queue.max_events unless you have aggressive disk-usage alerts, and monitor both queue.events and free space on the queue path. Most importantly, rehearse the failure: kill Logstash during a burst, restart it, and verify that your destination receives every event. Bursty workloads are inevitable; silent data loss does not have to be.