Logstash Persistent Queues: Keeping Events Safe When the Process Dies

Logstash normally holds events in memory as they move from input to output. If the process stops abruptly—an out-of-memory kill, a host reboot, a bad deploy—those in-flight events disappear. Downstream Elasticsearch clusters end up with time-series gaps that are hard to backfill. The fix is not more RAM; it is changing where events wait.

Persistent queues (PQs) move the buffer from RAM to disk. When enabled, Logstash writes every incoming event to a queue file on the local filesystem before a filter or output touches it. If the process dies, the events are still on disk. On restart, Logstash reads them back and continues processing. The guarantee is at-least-once delivery: an event might be processed twice after a crash, but it should not vanish.

How persistent queues work and how to enable them

A PQ behaves like a write-ahead log. Logstash appends events to page files—fixed-size chunks of disk storage. When a page fills, Logstash creates a new one. A separate checkpoint file records which pages are fully written and which events have been acknowledged by outputs. The checkpoint is what lets Logstash know where to resume after an unclean shutdown.

Because pages are written sequentially, disk throughput matters more than random I/O. PQ settings live in logstash.yml, not in the pipeline .conf file, and you must restart Logstash after changing them. These settings apply to Logstash 7.x and 8.x; confirm exact defaults against your specific version.

queue.type: persisted
queue.page_capacity: 512mb
queue.max_bytes: 8gb
queue.checkpoint.interval: 1000

queue.type: persisted activates the feature. queue.page_capacity controls the size of each individual page file; larger pages reduce file-system overhead but increase the amount of data that may be replayed after a crash. queue.max_bytes caps the total on-disk footprint. When the cap is reached, inputs block—Logstash applies back-pressure rather than consuming all available disk. queue.checkpoint.interval forces a checkpoint write after that many events are acknowledged, balancing durability against write amplification.

A crash test you can run in staging

The only way to trust a durability feature is to break it on purpose. The following example uses a file input, a GeoIP filter, and an Elasticsearch output. Run every command from a shell on the Logstash host; you need write access to the Logstash data directory and read access to the pipeline config.

First, create a minimal pipeline. Save this as /etc/logstash/conf.d/syslog-geoip.conf:

input {
  file {
    path => "/var/log/test-syslog.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  geoip {
    source => "[client_ip]"
  }
}

output {
  elasticsearch {
    hosts => ["https://es-host:9200"]
    index => "syslog-test-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "${ES_PASSWORD}"
  }
}

Generate a known volume of events. The exact count matters because you will compare it later:

# On the Logstash host, as a user who can write to the input path
for i in $(seq 1 1000); do
  echo "client_ip=8.8.8.8 message=test-$i" >> /var/log/test-syslog.log
done

Start Logstash and confirm it is reading:

# Run as the logstash service user or via systemd
sudo systemctl start logstash
curl -s -u elastic:${ES_PASSWORD} "http://localhost:9600/_node/stats/pipeline?pretty" | grep -A5 '"events"'

While events are flowing, kill the process hard:

# Find the main Logstash PID and send SIGKILL
sudo pkill -9 -f logstash

Restart Logstash and query Elasticsearch for the document count:

sudo systemctl start logstash
curl -s -u elastic:${ES_PASSWORD} "es-host:9200/syslog-test-*/_count" | jq '.count'

If persistent queues are working, the count should approach 1000 once the queue drains. If you see a large shortfall, the queue was not enabled or the data directory was not writable.

What persistent queues will not fix

PQs trade latency and disk for durability. Every event is written to disk before processing, so throughput drops compared to the in-memory queue. Under sustained heavy load, the disk can become the bottleneck.

More importantly, a PQ is not a cure for a slow downstream. If Elasticsearch is unavailable or chronically slow, the queue grows until it hits queue.max_bytes. At that point inputs block, but the disk is full and the pipeline is stalled. You still need back-pressure handling, dead-letter queues, or output-level retry tuning.

Upgrades add another wrinkle. Changing queue settings or upgrading across major versions can require draining the queue first. Always read the release notes before deploying; a half-drained queue from an older format may not replay cleanly.

Monitoring and next steps

Once PQs are enabled, watch the queue depth through the monitoring API:

curl -s -u monitoring_user:${MON_PASSWORD} "http://localhost:9600/_node/stats/pipeline" | jq '.pipelines.main.queue'

Look for events_in_queue and queue_size. A queue that stays near zero means the pipeline is keeping up. A queue that climbs steadily means outputs are the constraint, and you should tune batch sizes or add output workers before increasing queue.max_bytes.

Before you enable PQs in production, run the kill-and-restart test in staging with your real event shape. Verify that the document count matches and that restart time remains acceptable. Persistent queues only protect you if the disk they live on is as reliable as the data you are trying to save.