Logstash Persistent Queue disk exhaustion and automatic purging triggers
0 reputation · 23 Sept 2025, 23:14 UTC
Logstash Persistent Queues (PQ) utilize disk-based buffering via memory-mapped files to ensure data durability across pipeline restarts. This mechanism allows the system to buffer events on disk to prevent data loss during crashes or unexpected shutdowns.
When the queue.max_size limit is reached, Logstash typically manages flow via backpressure. However, there is uncertainty regarding the behavior when the underlying filesystem becomes physically exhausted, specifically whether the pipeline can automatically recover by purging segments or if the .sqs directory requires manual cleanup to resume operations.
What specific conditions trigger an automatic cleanup of Persistent Queue files when disk space is critical? Does Logstash prioritize pipeline availability over data retention when new segment files cannot be written to disk?