ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Logstash persistent queues write events to disk before processing, absorbing traffic spikes and surviving downstream outages without data loss. Learn configuration, verification, and the trade-offs of disk-backed buffering.
Logstash's default in-memory queue loses events during traffic spikes and crashes. Learn how persistent queues write to disk first, with a concrete configuration and verification steps.
Persistent queues turn Logstash from an in-memory pipe into a durable on-disk buffer. Here is how they work, how to configure them, and how to verify they actually prevent data loss during a crash.
A crash can erase events waiting in Logstash's default in-memory queue. Persistent Queues move that buffer to disk. This architecture note covers the smallest safe configuration, trust boundaries, operational checks, and failure modes.
Use Logstash Persistent Queues to keep events on disk when the process crashes, preventing data loss. Learn how to enable, verify, and monitor PQ, plus the trade‑offs and pitfalls.
Logstash Persistent Queues (PQ) utilize disk-based buffering via memory-mapped files to ensure data durability across pipeline restarts. This mechanism allows the system to buffer events on disk to prevent data loss during crashes or unexpected shutdowns. When the queue.max_size limit is reached, Logstash typically manages flow via backpressure. However, the
Migrating a small Logstash application without downtime typically requires utilizing persistent queues to buffer incoming data during a rolling update. While setting queue.type: persistent ensures that events are written to disk and survive restarts, the behavior under high concurrency remains complex. n When multiple worker threads are configured to process
logstash