Handling Downstream Outages with Logstash Persistent Queues

When a downstream destination like Elasticsearch becomes unavailable or slows down during a traffic spike, Logstash's default in-memory queue fills up quickly. Once full, Logstash blocks its input threads, creating back-pressure that can cause upstream sources to drop events or crash. Persistent queues solve this by offloading the buffer to disk, allowing Logstash to continue ingesting data during outages without losing events already accepted by the pipeline.

The Mechanics of Disk-Based Buffering

In a standard pipeline, events move from the input stage to the filter and output stages via memory. A persistent queue changes this by serializing incoming events to disk pages before they are processed. This ensures that if the Logstash process restarts or the output destination fails, the events remain stored on disk until they can be successfully delivered.

This mechanism provides "at-least-once" delivery for the output stage. It is important to note that this does not protect against events dropped by a misconfigured filter or bugs within an input plugin; it specifically protects the hand-off between the input and the rest of the pipeline.

Configuring the Queue

Persistent queue settings are global and must be defined in logstash.yml. They cannot be set within individual pipeline configuration files. Changes to these settings require a full restart of the Logstash service.

# logstash.yml
queue.type: persisted
queue.page_capacity: 64mb
queue.max_bytes: 4gb
queue.checkpoint.writes: 1024
path.queue: /var/lib/logstash/queue

• queue.type: Set to persisted to enable disk-based buffering.
• queue.page_capacity: The size of each disk segment. Larger pages can reduce I/O overhead but may increase the time it takes to drain the queue.
• queue.max_bytes: The hard limit for disk usage. If this limit is reached, Logstash will block input threads to prevent the disk from filling completely.
• queue.checkpoint.writes: Determines how often the read/write positions are flushed to disk. Lower values increase durability but increase disk I/O.

Execution Note: Ensure the user running the Logstash process (e.g., the logstash user) has full read/write permissions for the directory specified in path.queue. Run the service using the appropriate system user:

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d

Example: Protecting a File-to-Elasticsearch Pipeline

Consider a pipeline that parses application logs and sends them to Elasticsearch. Without a persistent queue, a 10-minute Elasticsearch outage during a high-traffic period would lead to lost logs or blocked inputs.

# /etc/logstash/conf.d/app_logs.conf
input {
  file {
    path => "/var/log/app/*.log"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/sincedb"
  }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}" }
  }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logs-app-%{+YYYY.MM.dd}"
  }
}

With queue.type: persisted enabled in logstash.yml, Logstash writes these parsed events to /var/lib/logstash/queue before attempting to send them to Elasticsearch. If the output fails, the events accumulate on disk and are replayed sequentially once the connection is restored.

Trade-offs and Limitations

Disk-based queuing is not a "free" upgrade. It introduces increased disk I/O and slightly higher latency compared to memory-only queues. Because the queue relies on the local filesystem, you must monitor disk space closely. If queue.max_bytes is set too high, you risk filling the system partition; if set too low, you will trigger back-pressure prematurely.

Verification Steps

To verify the configuration is working as expected:

1. Check Filesystem: Navigate to the path.queue directory. You should see .data files appearing as events are processed.
2. Simulate Failure: Temporarily stop the Elasticsearch service. Observe that Logstash continues to accept input and that the queue.page_count metric increases via the Logstash monitoring API.
3. Verify Recovery: Restart Elasticsearch. The .data files in the queue directory should decrease in size or disappear as the events are drained and delivered.