Choosing a Logstash Queue Type to Prevent Event Loss

Decision and Constraints

You need to guarantee that no events are lost when Logstash restarts or when traffic spikes temporarily exceed processing capacity. The decision must respect three practical constraints:

• Limited disk I/O budget – the queue should not overload slow storage.
• Acceptable latency increase – some extra delay is tolerable if it prevents loss.
• Operational simplicity – setup and monitoring should be straightforward.

Option Comparison

Queue Type	Durability	Throughput Impact	Disk Usage	Setup Complexity
none (in‑memory)	Low – events held only in RAM are lost on crash or kill	None – no disk I/O	None	Minimal – default configuration
persisted (file‑based)	High – queue pages survive crashes; events are replayed on restart	Moderate – each enqueue/dequeue writes to disk; impact depends on storage speed	Proportional to configured queue size (page files grow as needed)	Requires a writable directory, proper filesystem permissions, and optional tuning of page capacity and max bytes
persisted (embedded)	High – same durability as file‑based	Moderate – same I/O characteristics	Same as file‑based	Same as file‑based

Trade‑offs

The in‑memory queue offers the lowest latency and zero disk cost, but any abrupt termination of Logstash results in the loss of all buffered events. This risk is unacceptable when you must guarantee no loss during restarts or traffic bursts.

The persisted queue adds durability by writing queue pages to disk. It protects events across crashes and allows the pipeline to absorb short‑term spikes without dropping data. The trade‑off is increased I/O, which can reduce throughput on slow storage and requires monitoring of disk space to avoid hitting queue.max_bytes, at which point Logstash will block input plugins and create back‑pressure.

If your storage can sustain the required IOPS (e.g., a fast SSD or local NVMe) and you can allocate a reasonable amount of space (e.g., 1 GB), the persisted queue satisfies the durability goal while keeping latency impact within acceptable bounds.

Concrete Implementation

Enable a persisted queue with a 1 GB capacity on a fast SSD by adding the following to logstash.yml (run with the Logstash process user, which must have write permission to the queue path):

queue.type: persisted
queue.path: /var/lib/logstash/queue
queue.page_capacity: 64mb
queue.max_bytes: 1073741824

Explanation:

• queue.type: persisted activates the file‑based queue.
• queue.path specifies the directory where page files are stored; ensure the directory exists and is owned by the Logstash user.
• queue.page_capacity controls the size of each page file; 64 MiB is a common default that balances I/O granularity and metadata overhead.
• queue.max_bytes caps the total queue size; once reached, Logstash will pause input plugins.

Validation and Verification

To confirm that the persisted queue is working and that no events are lost:

1. Start Logstash with the above configuration.
2. Inject a steady stream of events using the generator plugin (e.g., input { generator { count => 1000000 } }) feeding a simple stdout or file output.
3. While the pipeline is under load, stop Logstash abruptly (e.g., kill -9 <pid>).
4. Restart Logstash.
5. Check the output count; it should match the number of events generated before the stop.
6. Alternatively, monitor the queue directory: page files (queue.page_*) should appear and grow as events accumulate.
7. Use the Logstash REST API to observe queue metrics:

curl -s http://localhost:9600/_node/stats/pipeline?pretty

Look for fields such as queue.events_in_queue and queue.events_dequeued. Under load, events_in_queue should fluctuate but never drop to zero when the pipeline is stopped and restarted; after restart, the dequeued count should continue from where it left off.

Limitations and Operational Checks

• If the storage behind queue.path has low IOPS (e.g., a network‑mounted drive with high latency), the persisted queue can become a bottleneck, increasing end‑to‑end latency and potentially causing input back‑pressure.
• Regularly monitor disk usage on the queue partition. If usage approaches queue.max_bytes, Logstash will block inputs; consider increasing the limit or adding more storage.
• Ensure the filesystem supports reliable writes (e.g., ext4, xfs) and that the mount options do not disable sync or enable aggressive caching that could compromise durability.

By following the configuration and verification steps above, you can achieve a durable Logstash pipeline that meets the constraints of limited I/O budget, acceptable latency, and operational simplicity.