DogStatsD Buffer Limits and High-Cardinality Memory Pressure
0 reputation · 13 Aug 2020, 01:38 UTC
DogStatsD Resource Management
The Datadog Agent implements a fixed-size buffer for DogStatsD metrics to prevent unbounded memory growth during throughput spikes. While the Agent is designed to drop packets when this buffer reaches capacity, the local aggregation map remains subject to memory pressure based on the volume of unique metric/tag combinations.
Cardinality Constraints
High-cardinality tags increase the size of the aggregation map held in memory between flush intervals. This creates a tension between the fixed UDP buffer limit, which prevents system crashes, and the dynamic memory requirements of the aggregation process.
When managing high-throughput environments, it is unclear how the Agent prioritizes memory allocation between the incoming packet buffer and the internal aggregation map during extreme cardinality events.
- Does the Agent implement a hard memory cap for the aggregation map similar to the fixed UDP buffer?
- How does the flush interval specifically influence the peak memory footprint when processing high-cardinality tags?
datadog