Diagnosing Grafana Alert Rule Evaluation Failures in 9.x and 10.x

Recognizable condition

In Grafana 9.x and 10.x, an alert rule may appear to evaluate correctly but never trigger a notification. The rule preview shows “No data” or an evaluation error, and the alert state stays at “Normal” despite the underlying metric crossing the threshold.

Common causes and quick diagnostic table

Cause	Typical symptom in the rule preview	Initial check
Datasource unreachable	“No data” or “datasource error”	Test the datasource connection from Configuration → Data Sources
Invalid query syntax or missing metric/label	Parsing error or “no series returned”	Open the rule, click “Test Rule” and inspect the query output
Evaluation interval longer than the for duration	Rule stays in “Pending” forever, never reaches “Firing”	Compare the rule’s evaluation interval with its for clause
Contact point misconfigured or disabled	Rule preview shows correct firing state but no notification arrives	Send a test notification from Alerting → Contact Points
Active silence or mute timing	Rule fires but notifications are suppressed	Check Alerting → Silences and Mute Timings for matching rules

Ordered checks

1. Verify datasource health
   • Navigate to Configuration → Data Sources, select the Prometheus/Loki datasource used by the rule.
   • Click Test Connection. A successful response shows “Data source is working”.
   • If the test fails, note the error (e.g., timeout, authentication failure) and address network or credential issues.
2. Test the alert rule query
   • Open the alert rule (Alerting → Rules), click Test Rule.
   • Examine the preview pane:
     • If you see an error message, correct the PromQL/LokiQL syntax or ensure the referenced metric/label exists.
     • If the preview returns “No data”, verify that the metric is being scraped at the expected interval and that the datasource time range matches the rule’s evaluation window.
3. Check evaluation interval vs. for duration
   • In the rule settings, locate Evaluate every (interval) and For duration.
   • Ensure the interval is less than or equal to the for value; otherwise the rule can never satisfy the sustained condition.
   • Adjust either setting so that interval ≤ for.
4. Validate contact point configuration
   • Go to Alerting → Contact Points, select the contact point used by the rule’s notification policy.
   • Click Send test notification. Confirm receipt via the configured channel (email, Slack, etc.).
   • If the test fails, correct the integration details (webhook URL, API key, SMTP credentials).
5. Look for silencing or muting
   • Open Alerting → Silences and Alerting → Mute Timings.
   • Delete or edit any silence/mute that matches the rule’s labels or falls within its active period.

Fixes tied to findings

• Datasource unreachable: Correct the URL, credentials, or proxy settings; re‑test the connection.
• Invalid query: Edit the rule’s query, use the metric explorer to confirm series exist, and save.
• Interval > for: Reduce the evaluation interval (e.g., from 2m to 1m) or increase the for duration (e.g., from 1m to 2m).
• Contact point error: Fix the webhook URL, token, or SMTP settings and resend the test.
• Silence/mute: Remove the silencing rule or adjust its time range.

Escalation criteria

If after completing the ordered checks the alert rule still does not fire or notifications are not received:

• Collect the rule’s JSON definition (Alerting → Rules → … → View JSON) and the datasource proxy logs.
• Verify that the Grafana server can reach the datasource host from the server OS (e.g., curl -v http://:9090/-/ready).
• Check Grafana’s internal logs (/var/log/grafana/grafana.log) for messages containing “alerting”, “eval”, or “notification”.
• If the logs indicate a plugin or backend failure, open a support ticket with Grafana Labs or your platform provider, attaching the rule JSON, datasource config, and relevant log excerpts.

Note: This guide assumes Grafana 9.x or 10.x with unified alerting. Legacy alert rules stored in the pre‑8.0 database may require a migration step before these checks apply.