DogStatsD Buffer Limits and High-Cardinality Memory Pressure

Answer to the Core Questions

1. Does the Agent impose a hard memory cap on the aggregation map?

• Yes, starting with Agent 7.40 (and 6.40 for older releases) the dogstatsd_cardinality_limit setting provides a hard limit on the number of unique series that can be kept between flushes.
• When the limit is reached, the Agent evicts least‑recently‑used series at the next flush. Before 7.40 there was no such cap; the map could grow unbounded until the next flush.

2. How does the flush interval influence peak memory?

• The UDP receive buffer (default ~8 MB, configurable via dogstatsd_buffer_size) is fixed and independent of flush timing.
• The aggregation map grows with the number of unique series that arrive between flushes. A longer dogstatsd_flush_interval (default 10 s) lets more series accumulate, so peak memory is roughly unique_series × entry_overhead plus the UDP buffer.
• Halving the flush interval typically halves the peak memory footprint for a steady high‑cardinality load, assuming the cardinality limit is not hit.

Confirmed Facts (based on current Agent releases)

• UDP buffer size defaults to 8 MB; packets are dropped when it’s full, protecting the OS socket buffer.
• dogstatsd_cardinality_limit defaults to 100,000 unique series per flush interval.
• Eviction occurs only at flush time; packets are not dropped due to cardinality pressure.
• Containerized Agents must have net.core.rmem_max ≥ dogstatsd_buffer_size to avoid early kernel drops.

Practical Steps for Your Environment

1. Verify Agent version:

   datadog-agent version

   Ensure it’s ≥ 7.40 (or 6.40 for older series).
2. Check effective configuration:

   datadog-agent configcheck | grep -E "dogstatsd_(buffer_size|flush_interval|cardinality_limit)"

   Note the current values.
3. Monitor runtime metrics:

   datadog-agent status | grep -A5 "dogstatsd"
   # look for packets_dropped, metrics_by_context, series_count

4. Run a controlled cardinality test (optional but useful for verification):

   for i in $(seq 1 200000); do
     echo "test.metric:1|c|#tag:$i" | nc -u -w0 127.0.0.1 8125
   done

   Watch RSS in docker stats or ps -o rss and confirm that memory growth stops once the limit is hit.
5. Adjust if needed:
   • Reduce dogstatsd_flush_interval to lower peak memory for steady high‑cardinality traffic.
   • Lower dogstatsd_cardinality_limit if you’re hitting the limit but can tolerate dropping low‑volume series.

Missing Diagnostic Detail

To fine‑tune the recommendation, could you share your current dogstatsd_cardinality_limit value? If it’s set to the default or higher than your observed series count, you may want to lower it or adjust the flush interval.