Question
Which logic governs notification behavior for high-cardinality Multi Alerts?
Kavi Lane
0 reputation · 24 Aug 2020, 22:52 UTC
105.1K views1
Loading…
0 reputation · 24 Aug 2020, 22:52 UTC
Datadog monitors configured for Multi Alerts evaluate thresholds against each unique combination of tags independently. While this allows for granular visibility, high-cardinality tags can result in a significant volume of individual notifications when multiple groups cross the threshold simultaneously.
When utilizing complex boolean logic in monitor queries, it becomes difficult to isolate the specific metric group responsible for a trigger. There is a need to understand how the system handles the transition between trigger and recovery states when multiple groups are flapping near the threshold limit.
In a Datadog Multi Alert monitor each unique tag combination is evaluated independently. Notifications are generated per group based on the monitor’s evaluation delay and renotify interval; there is no built‑in mechanism to aggregate notifications across different groups within the same monitor.
datadogevaluation_delay) is applied to each group before a trigger or recovery notification is sent.renotify_interval), Datadog resends a notification for a group every interval while the group remains in the trigger state.Because each tag combination is treated as its own alert, high‑cardinality tags can cause many groups to cross the threshold simultaneously, resulting in a proportional increase in notifications. The system does not combine these events into a single notice unless you restructure the monitor (e.g., using a separate monitor with broader tags or using monitor‑wide notification settings).
evaluation_delay and renotify_interval values.by clause (or the group by fields) to see which tags define the groups.If you have configured any monitor‑wide notification or aggregation settings (e.g., using a Composite monitor or enabling “Notify on no data” with aggregation), the recommendation could change. Please confirm whether such settings are active for this monitor.
Use comments to ask for clarification. Post a solution as an answer.
6,130 reputation · 25 Aug 2020, 08:56 UTC
An often-overlooked interaction in high-cardinality Multi Alert monitors is how no-data handling per group can surface independently of threshold breaches. When a tag group stops reporting metrics entirely, Datadog applies the monitor's no-data action - such as firing an alert or transitioning to OK - based on the configured no-data duration. This means sparse series across many tag combinations can generate a flood of no-data alerts that compound the notification volume already amplified by threshold crossings. In practice, teams often set a longer no-data duration or suppress no-data for high-cardinality groups to keep alert noise manageable. If you're observing unexpected alerts during periods of metric silence, checking the monitor's no-data settings alongside the evaluation delay can reveal whether group-level data gaps are the contributing factor; this behavior is typically consistent across Datadog SaaS monitors but should be confirmed against your monitor's specific no-data action and evaluation delay configuration.