Question
Prometheus 2.40 WAL segment size transition for low-traffic disk reduction
Kavi Indigo
0 reputation · 26 Aug 2024, 21:08 UTC
36.6K views
Loading…
0 reputation · 26 Aug 2024, 21:08 UTC
For a Prometheus instance handling roughly 10 samples per second, the goal is to lower the write-ahead log segment size from the long-standing 128 MiB default to a smaller value such as 8–16 MiB, using the --storage.tsdb.wal-segment-size flag introduced in version 2.40. The intent is to reduce the storage footprint and limit the amount of unreclaimed data during short scrape intervals without changing query behavior.
The uncertainty lies in how this configuration change interacts with existing retention boundaries defined by --storage.tsdb.retention.time and --storage.tsdb.retention.size, and whether the resulting increase in segment count and truncation frequency introduces unacceptable fsync or CPU overhead on the host. Because the flag is unavailable before 2.40, any mixed-version rollout or rollback plan must account for startup failures on older binaries, and practical verification of actual segment sizes and disk usage over a stable low-traffic period is required before production adoption.
What is the safest way to evaluate the trade-off between reduced disk usage and higher WAL checkpoint frequency for a low-traffic workload? Which retention settings should be re-examined before committing to the smaller segment size?
33775 reputation · 26 Aug 2024, 23:35 UTC
To evaluate the trade-off between disk usage and checkpoint frequency, the safest approach is a staged rollout using a canary instance or a non-production environment, monitoring the prometheus_tsdb_wal_segments_count metric and filesystem metadata overhead. Because the --storage.tsdb.wal-segment-size flag was introduced in version 2.40, any environment using older binaries will fail to start if this flag is present.
In a low-traffic environment (e.g., 10 samples/s), the default 128 MiB segment size can lead to a disproportionately large amount of data sitting in the Write-Ahead Log (WAL) before it is compacted into a TSDB block. Reducing this to 8–16 MiB forces more frequent rotations and checkpoints.
prometheus/wal directory. For most modern filesystems (ext4, XFS), the overhead of managing a few extra segments per day is negligible at this scale.fsync frequency, the low sample rate means the actual data being flushed is minimal, making CPU spikes unlikely.It is important to note that --storage.tsdb.wal-segment-size operates independently of your retention boundaries. However, you should review the following to ensure overall disk stability:
--storage.tsdb.retention.size: If you are reducing WAL size to save disk space, ensure your total retention size limit is not so tight that the increased frequency of block creation (resulting from smaller WALs) triggers aggressive deletion of older blocks.--storage.tsdb.retention.time: Ensure the retention window is aligned with your business needs; the WAL size change only affects the "head" of the data, not the long-term storage.To implement this change safely, follow these steps:
--storage.tsdb.wal-segment-size=16MB (or 8MB) to the startup arguments.data/wal/ directory. New segments should match the configured size.prometheus_tsdb_wal_segments_count metric to ensure the rotation rate is stable and not causing excessive I/O wait.Diagnostic Detail Needed: Please specify the underlying storage medium (e.g., SSD, HDD, or Network Attached Storage). If using slow network storage, the increased frequency of fsync calls during segment rotation may be more impactful than on local NVMe drives.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.