Set Up Grafana Alerting with Prometheus: Panel Rules and Notifications

The problem: dashboards do not page you

Dashboards show trends after you look at them. Alerts interrupt you when a Prometheus metric crosses a dangerous threshold. In Grafana 9.x and later, you can attach an alert rule directly to a dashboard panel instead of maintaining a separate Prometheus rule file. The useful takeaway: the rule only works if your panel query reduces to a single numeric value and your evaluation interval is shorter than the condition’s lookback window.

Prerequisites

• Grafana 9.x or later with the Alerting UI available. Alerting features and limits vary by edition and version; verify your deployment supports panel alerts and the notification channels you plan to use.
• A Prometheus data source added under Configuration → Data sources, reachable from Grafana, with PromQL enabled.
• A metric that can be reduced to a single number, such as CPU idle percentage or request error ratio.
• Editor or Admin permissions to save dashboard alerts and manage notification channels.

Step 1: Verify the Prometheus data source

Open Configuration → Data sources → Prometheus. Confirm the URL (for example, http://prometheus:9090) is reachable from the Grafana server. Click Save & Test. If the test fails, check network policies, firewalls, or authentication settings before continuing.

Step 2: Build a panel that returns a single value

Create or edit a dashboard panel. Alert conditions apply a reducer such as avg() to whatever the query returns, so the query itself should already be close to a single numeric series.

Example CPU query:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

This calculates the percentage of non-idle CPU time per instance. If you want one alert for the entire fleet rather than one per host, remove the by (instance) clause so the query returns a single series. If your query returns many unrelated series, the alert condition will average them together, which may hide the outlier you care about.

Step 3: Define the alert condition

In the panel editor, open the Alert tab and click Create Alert. Configure the condition:

• WHEN avg() OF query(A, 5m, now) IS ABOVE 80

Here, avg() reduces all series from query A to one number. The 5m lookback window means Grafana evaluates the average over the last five minutes. now anchors the evaluation to the current time.

Set the evaluation behavior:

• Evaluate every 1m
• For 3m (three consecutive periods)

The evaluation interval must be shorter than the condition’s time range. If you evaluate every 10 minutes but the condition looks back only 5 minutes, you will skip windows where the metric spiked and recovered. The pending period suppresses flapping: the rule must be continuously true for three minutes before it fires.

Step 4: Configure a notification channel

Go to Alerting → Notification channels and create a channel for Email, Slack, or PagerDuty. Provide the required credentials or webhook URL. Grafana masks sensitive fields after you save them, but you should still restrict access to the Grafana configuration. Return to the alert rule, expand the Notifications section, and attach the channel.

Step 5: Save and enable

Save the dashboard. The alert rule appears under Alerting → Alert rules. Grafana now evaluates it automatically according to the interval you set.

Expected checks and alert states

Open Alerting → Alert rules and inspect the rule. The state column tells you what Grafana decided on the last evaluation:

State	Meaning
Normal	Condition is false.
Pending	Condition is true, but the required consecutive periods have not elapsed.
Firing	Condition has been true long enough; notifications are sent.
No Data	The query returned no series.
Error	The data source is unreachable or the query failed.

Verifying the rule without guessing

1. Use the Test tab. In the alert rule editor, open the Test tab, select a recent time range where the metric exceeded the threshold, and confirm the rule would have entered a Firing state.
2. Force a known value. Temporarily replace the panel query with constant_vector(100). This returns the scalar 100 for every timestamp, guaranteeing the condition is met. Wait for the pending period to elapse and confirm the notification channel receives the alert. Risk: if you forget to revert the query, the alert will remain firing.
3. Check delivery logs. Look in the email inbox, Slack channel, or PagerDuty service for the alert payload.
4. Simulate an outage. Optionally stop Prometheus or block Grafana’s access to it. The rule should move to Error or No Data. Alerts will not fire until connectivity is restored. Restart Prometheus and confirm the rule returns to a data-driven state.

Recovery options

• Disable the rule. Toggle the rule off in Alerting → Alert rules. This stops evaluation and silences notifications without deleting your configuration.
• Delete the rule. Remove it from the panel or the Alert rules page if you no longer need it.
• Remove the notification channel. Only delete a channel after confirming no other rules depend on it.
• Fix a failing query. If the rule reports Error because the query “did not return a single numeric value,” add an explicit reducer such as avg() or sum() in the panel query itself.

Limitations to keep in mind

• Panel alerts evaluate the query as written. If the query returns multiple series, the condition reducer averages them unless you group or filter explicitly.
• The alert fires as a single unit for the whole query result. Per-instance routing requires label templating in the notification message, not separate rules.
• Alerting capabilities, retention, and channel types vary by Grafana edition and version. Verify your deployment before relying on a specific feature in production.
• When Prometheus is down, Grafana cannot invent data. Expect Error or No Data, not a firing alert.

Practical final check

After saving the rule, force a test with constant_vector(100), confirm the notification arrives, revert the original query, and then open Alerting → Alert rules to verify the Last evaluated timestamp is recent and the state reflects live data. If both are true, the alert pipeline is working end to end.