Configuring Persistent Queues in Logstash for Durable Event Processing

Desired outcome

Enable Logstash persistent queues so that in‑flight events are saved to disk and can be recovered after an unexpected shutdown or restart, without data loss.

Necessary prerequisites

• Logstash version 7.0 or later (persistent queues were introduced in 6.0 and stabilized in 7.x).
• Administrative access to the host where Logstash runs (to edit logstash.yml and restart the service).
• Sufficient free disk space on the volume that will hold the queue; a rule of thumb is at least twice the expected peak in‑flight event size.
• Backup of the current logstash.yml in case you need to revert.

Focused procedure

1. Identify the queue directory. Edit logstash.yml (usually located at /etc/logstash/logstash.yml) and set or verify path.queue. Example:

   # path.queue: /var/lib/logstash/queue
   path.queue: /mnt/logstash_queue

   Ensure the directory exists and is owned by the Logstash user (chown -R logstash:logstash /mnt/logstash_queue).
2. Enable the persisted queue type. Add or modify the following line in the same file:

   queue.type: persisted

   Optionally tune the queue characteristics (see table below).
3. Apply the change. Save the file and restart Logstash. On a systemd host:

   sudo systemctl restart logstash

   If you run Logstash from the command line, stop the current process (Ctrl+C or kill) and start it again with the same flags.
4. Verify that the persistent queue is active. Check the startup log for the line:

   [INFO ] logstash.agent - Persistent queue is enabled

   Then list the queue directory:

   ls -la $path.queue
   # Expected output includes:
   # queue.log  (the active page file)
   # checkpoint  (contains the last processed batch)

   The presence of queue.log and a checkpoint file indicates that events are being written to disk.
5. Test durability. 1. Send a few test events to a pipeline (e.g., via curl -XPOST 'http://localhost:5044' -d '{"msg":"test"}' if using Beats input). 2. While Logstash is still running, kill it abruptly to simulate a crash:

   sudo kill -9 $(pgrep -f logstash)

   3. Restart Logstash as in step 3. 4. Observe the output (e.g., Elasticsearch, stdout, or a file). The previously sent test events should appear after restart, confirming recovery from the persistent queue.

Expected checks

• Startup log contains Persistent queue is enabled.
• The path.queue directory shows a growing queue.log file and a checkpoint file that updates after each batch.
• After an abrupt stop and restart, events that were in flight before the stop appear in the downstream output.

Relevant recovery options (rollback)

Changing queue.type requires a full restart; there is no in‑place migration. To revert to the default in‑memory queue:

1. Restore the original logstash.yml (or comment out the queue.type line).
2. Stop Logstash.
3. Optionally, clear the queue directory to avoid stale checkpoint files interfering with the in‑memory queue (rm -rf $path.queue/*).
4. Start Logstash again.

Note: Removing the queue directory while Logstash is running will cause the pipeline to stall; perform the cleanup only when the service is stopped.

Limitations and practical verification

• Persistent queues increase I/O latency; monitor disk latency (iostat -x 1) and ensure the underlying storage can sustain the expected write rate.
• The default page_capacity (64 MB) and max_bytes (1 GB) are suitable for most workloads; adjust only after observing queue growth patterns.
• If the queue directory runs out of space, Logstash will block the pipeline and log warnings like Persistent queue is full. Prevent this by setting up disk‑usage alerts on the volume hosting path.queue.

Configuration reference table

Setting	Description	Typical value
queue.type	Queue implementation; persisted enables disk‑backed queue.	persisted
page_capacity	Maximum size of each page file before a new page is created.	64mb
max_bytes	Upper bound for total queue size on disk; older pages are dropped when exceeded.	1gb
checkpoint.writes	Number of batch checkpoints to keep; higher values increase durability but use more disk.	1024