Logstash Persistent Queues: When to Enable and How to Validate

Decision and Constraints

Enable Logstash Persistent Queues (PQ) when losing events on a crash or restart is unacceptable and your ingest throughput fits within the IOPS of a local disk. Disable PQ for pure high‑throughput ephemeral pipelines where the in‑memory queue’s sub‑millisecond latency and higher event rate are acceptable.

Constraints:

• Requires Logstash 5.0 or later (tested on 8.x).
• Each pipeline gets its own queue directory under path.queue (default data/queue).
• Maximum queue size set via queue.max_bytes (default 1 GB, minimum 1 MB, maximum 10 TB).
• Page size controlled by queue.page_capacity (default 64 MB).
• Do not share path.queue across pipelines or nodes — corruption will occur.

Comparison: In‑Memory vs Persistent Queues

Trade‑offs

PQ adds an fsync latency mitigated by checkpoint batching (queue.checkpoint.writes, default 2048 in 8.x). Disk saturation blocks inputs, so size the queue to leave headroom for OS page cache and other Logstash data — set queue.max_bytes ≤ 70 % of a dedicated disk. Heap is still used for batch processing (pipeline.batch.size); PQ does not replace output acknowledgements — use a dead‑letter queue (DLQ) for failed writes.

Concrete Configuration

Place the following in logstash.yml (or pipeline‑specific pipelines.yml) on a node with a dedicated SSD mounted at /var/lib/logstash/queue:

queue.type: persisted
queue.max_bytes: 4gb
queue.page_capacity: 64mb
queue.checkpoint.writes: 2048
queue.drain: true
path.queue: /var/lib/logstash/queue

Run this configuration as the Logstash service user (typically logstash) with write permissions on the queue directory. Verify the directory exists and is on a separate physical device to avoid contention with OS or other workloads.

Validation Pipeline

Create a minimal pipeline to prove durability:

input {
  stdin { }
}
output {
  stdout { codec => rubydebug }
}

Start Logstash with this pipeline, then feed a known number of events (e.g., 10 000 lines). While events are flowing, send kill -9 <pid> to the Logstash process (requires root or same‑user permission). Restart Logstash with the same pipeline and count events emitted to stdout. The count should match the input count, and each event’s @metadata.queue_id should be identical across the restart.

Risk: kill -9 terminates the process instantly; any in‑flight batches not yet checkpointed will be replayed, which is the intended behavior. Do not run this test on a production pipeline.

Monitoring and Alerts

Watch these metrics via the Node Stats API (GET /_node/stats) or Prometheus exporter:

• logstash.stats.queue.events — current queue depth
• logstash.stats.queue.bytes — bytes used
• OS disk latency: iostat -x 1 on the queue device (run as root or with sysstat permissions)

Alert when queue.bytes > 0.8 * queue.max_bytes or when disk await exceeds 10 ms consistently.

Limitations and Cautions

• PQ files are not encrypted at rest; encrypt the underlying volume if compliance requires it.
• PQ does not survive disk failure; pair with filesystem snapshots or replication for HA.
• Logstash 8.x changed default queue.checkpoint.writes from 1024 to 2048; verify version‑specific defaults before tuning.
• Setting queue.max_bytes too high can starve the OS page cache; keep ≤ 70 % of dedicated disk.

Verification Checklist

1. Confirm Logstash version: bin/logstash --version (requires 5.0+).
2. Check PQ enabled: Node Stats API shows queue.type: persisted.
3. Simulate crash: kill -9 <pid> during ingest; restart; count output events — should match input count.
4. Measure latency delta: compare stats.events.duration_in_queue with PQ on vs off using same workload.
5. Disk sizing test: write at peak rate until queue.bytes hits max_bytes; verify back‑pressure blocks input.