Keeping Logs Safe: Using Logstash Persistent Queues for At‑Least‑Once Delivery
Learn how Logstash persistent queues protect in‑flight events during restarts, with a concrete configuration example, verification steps, and trade‑offs to consider.
ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Learn how Logstash persistent queues protect in‑flight events during restarts, with a concrete configuration example, verification steps, and trade‑offs to consider.
Learn how to spot DLQ‑related event loss, diagnose the root cause, apply fixes, and know when to escalate.
Learn how to use Logstash Persistent Queues (PQ) to prevent data loss during crashes or downstream outages by moving buffers from RAM to disk.
Guide to selecting between in‑memory and persisted Logstash queues based on durability, I/O impact, and operational constraints.
Learn how to enable Logstash's Dead Letter Queue, capture failing events, and safely reprocess them after fixing pipeline issues.
When parsing structured logs in Logstash, deciding between Grok and Dissect hinges on format stability, performance, and complexity. This guide compares both, presents trade‑offs, and shows a concrete Dissect pipeline with validation steps.
Persistent queue behavior changes when a Logstash pipeline moves from the default in-memory queue to a persisted disk-backed queue. The documented change shifts memory pressure to disk, alters backpressure when queue.max_bytes is reached, and changes restart semantics to replay unacknowledged pages. Version assumptions matter because queue.type names and def
When using the Logstash HTTP input plugin, the administrator wants to verify that the endpoint is not reachable from untrusted networks. The plugin binds to 0.0.0.0 by default and does not require authentication unless ssl or user/password options are explicitly set, leaving the pipeline open to anyone who can reach the port. In environments where the host s
Goal Determine whether Logstash can expose a per‑pipeline shutdown timeout that can be adjusted while the instance is running, rather than relying on the global pipeline.shutdown_timeout setting in logstash.yml or the pipeline configuration file. Constraints & Uncertainty The current implementation treats pipeline.shutdown_timeout as a global value, appl