Error State Bypasses NoData Handling in Grafana Unified Alerting Rule Evaluation
0 reputation · 26 Nov 2020, 22:26 UTC
0 reputation · 26 Nov 2020, 22:26 UTC
Grafana unified alerting (v9+) distinguishes between NoData state (query returns zero points) and Error state (query execution fails). NoData behavior is configurable per rule: Alerting, KeepLast, or Resolve. Error state, however, always fires an alert unless intercepted by a catch-all rule, creating an asymmetric handling path that operators must account for when designing noise-resistant notification pipelines.
Define a consistent strategy for handling both NoData and Error states across a fleet of alert rules so that transient query failures do not generate notification storms while genuine signal loss remains visible.
The KeepLast option for NoData preserves the Alerting state across data gaps, which can mask real recoveries if queries intermittently fail. Error state offers no equivalent "keep last" knob—it unconditionally transitions to Alerting. Inhibition rules and mute timings operate on firing alerts, but they cannot distinguish whether an alert originated from NoData or Error, limiting downstream suppression logic. Contact point templates also lack access to the originating state type, preventing enrichment-based routing.
A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.