Using Logstash Persistent Queues to Prevent Data Loss on Restart

When a Logstash node crashes or is restarted for maintenance, any events that have been accepted by an input plugin but not yet sent to the output are lost. This can break downstream analytics or cause gaps in monitoring data. Persistent queues solve this by writing each event to disk before acknowledging the input, guaranteeing at‑least‑once delivery even after a hard kill.

Why persistent queues matter

Logstash pipelines are often treated as stateless processors, but the reality is that inputs can ingest data faster than outputs can ship it, especially during network hiccups or output back‑pressure. Without durability, a sudden node failure means those in‑flight events disappear. Persistent queues act as a write‑ahead log (WAL) that survives process termination, letting Logstash replay the stored events on startup before accepting new ones.

How the feature works

When queue.type: persisted is set, Logstash creates a directory defined by path.queue (default /var/lib/logstash/queue). Each incoming event is serialized and appended to a series of page files; the input plugin is only acknowledged after the write succeeds. On startup, Logstash replays all persisted pages in order, reconstructing the exact event stream that was in flight at the moment of shutdown. The mechanism works with every input plugin because the queuing happens inside the core pipeline, not inside the plugin itself.

Configuration steps

1. Edit logstash.yml (usually located at /etc/logstash/logstash.yml) and add or modify:
• queue.type: persisted
• queue.page_capacity: 64mb    # size of each page file (optional)
• queue.max_bytes: 1024mb    # total queue size limit (optional)
2. Ensure the Logstash OS user can read and write the queue directory:
• sudo chown -R logstash:logstash /var/lib/logstash/queue
• sudo chmod -R 750 /var/lib/logstash/queue
3. Restart Logstash to apply the changes:
• sudo systemctl restart logstash

If you are running Logstash from a tarball or Docker, adjust the paths accordingly; the principle remains the same.

Worked example and verification

1. Create a simple pipeline config test.conf:

input { stdin { } }
output { stdout { codec => rubydebug } }

2. Start Logstash with this pipeline:

bin/logstash -f test.conf --path.settings /etc/logstash

3. In another terminal, type a few lines of text and press Enter after each (e.g., apple, banana, cherry). You should see each line echoed back in rubydebug format.
4. While Logstash is still running, simulate a crash:

kill -9 <pid-of-logstash-process>

5. Restart Logstash with the same command.
6. Immediately after startup, you should see the previously typed lines appear again in the output before the prompt returns for new input. This demonstrates that the events were persisted and replayed.
7. Inspect the queue directory to confirm the WAL files:

ls -lh /var/lib/logstash/queue

You will notice one or more queue.page.* files whose size grows with the number of events you sent and shrinks after they have been replayed.

Permissions and risks: The commands above must be run as a user with sufficient privileges to manage the Logstash service and access the queue directory (typically the logstash user or via sudo). If the queue directory runs out of space because downstream outputs are stalled, Logstash will block further inputs once queue.max_bytes is reached, potentially causing back‑pressure upstream. Monitor disk usage and set alerts on the queue directory size.

Trade‑offs and limitations

• Performance: Persistent queues add a modest throughput penalty (typically 5‑15 %) and a few milliseconds of latency due to sequential disk writes. On fast local SSDs this impact is usually negligible.
• Disk requirements: You must provision enough storage for queue.max_bytes plus a safety margin. Network‑mounted filesystems (NFS, CIFS) can become bottlenecks and are discouraged for production queues.
• Startup time: On very large queues, replay can take several seconds or minutes during which Logstash does not accept new inputs. Plan maintenance windows accordingly.

Actionable closing

If your Logstash deployment cannot tolerate lost events during a node restart, enable persistent queues today. Start with the default page capacity, monitor the queue directory size, and verify the replay behavior with a simple stdin/stdout test as shown above. Adjust queue.page_capacity and queue.max_bytes based on your event rate and acceptable disk usage, and always place the queue on a low‑latency local SSD to keep the performance impact minimal.