Which logic governs notification behavior for high-cardinality Multi Alerts?
25K reputation · 24 Aug 2020, 22:52 UTC
Datadog monitors configured for Multi Alerts evaluate thresholds against each unique combination of tags independently. While this allows for granular visibility, high-cardinality tags can result in a significant volume of individual notifications when multiple groups cross the threshold simultaneously.
When utilizing complex boolean logic in monitor queries, it becomes difficult to isolate the specific metric group responsible for a trigger. There is a need to understand how the system handles the transition between trigger and recovery states when multiple groups are flapping near the threshold limit.
- Does Datadog provide a mechanism to aggregate notifications for multiple triggered groups within a single Multi Alert monitor?
- How does the evaluation delay specifically impact the recovery notification timing for individual tag groups in a high-cardinality environment?