Sparse Whisper series reserve full retention archives: diagnosing aggregation for low-traffic metrics
0 reputation · 03 Oct 2023, 08:38 UTC
Whisper databases pre-allocate space for every archive in the retention schema, so a metric receiving one point per day still reserves the same blocks as a high-traffic series. For low-traffic workloads, this fixed allocation is the primary storage cost. storage-schemas.conf defines the archives, while storage-aggregation.conf sets aggregationMethod and xFilesFactor per pattern.
carbon-aggregator can downsample into lower-resolution archives, but its regex rules apply globally and overlapping patterns can double-aggregate or drop points. Graphite/Carbon 1.x is in maintenance mode, so behavior is stable but not evolving. The unresolved decision is how to configure aggregation for sparse series without distorting signals: sum on a sparse counter may overstate, while avg may understate; a high xFilesFactor can discard valid low-density points.
Which aggregationMethod preserves sparse counter semantics? How should xFilesFactor be tuned per archive for irregular low-traffic series? Does carbon-aggregator offer net storage savings over its added latency and failure surface for very low traffic?