Setting Up a CPU Alert in Grafana Unified Alerting: From PromQL to Slack

If your Grafana dashboards show CPU usage but nobody gets paged when a server pegs at 100%, the dashboard is decoration. Grafana 8.0 replaced the old dashboard-panel alerting with unified alerting, a standalone rule engine that evaluates queries on a schedule and routes notifications through reusable contact points. This post walks through one complete, concrete setup: a Prometheus-based CPU alert that fires when a host's idle CPU drops too low, delivered to a Slack channel.

Assumptions: Grafana 8.0 or newer (the examples reflect the 9.x/10.x UI), a Prometheus datasource already configured, and node_exporter metrics available.

Why unified alerting is worth the switch

The legacy engine tied each alert to a graph panel, which meant deleting a panel could silently delete its alert. Unified alerting decouples rules from dashboards: rules live in their own section, can query multiple datasources in one rule, and can be exported as JSON for version control. You also get a Test Rule button that evaluates the rule immediately without sending real notifications — useful for validating expressions before they page anyone.

Step 1: Confirm unified alerting is enabled

On Grafana 9.x and later, unified alerting is the default. On 8.x, check grafana.ini (or the environment variable equivalent) on the Grafana server — this requires filesystem access on the host running Grafana:

[unified_alerting]
enabled = true

Restart Grafana after changing it (systemctl restart grafana-server on systemd-based Linux). Verify by opening Alerting → Alert rules in the UI; if you see the modern rule editor rather than a prompt about legacy alerting, you are set. You can also confirm rule evaluations later in /var/log/grafana/grafana.log.

Step 2: Write the alert rule

In Alerting → Alert rules → New alert rule, give the rule a name like host-cpu-saturated. For the query, use the classic node_exporter CPU expression — the 5-minute rate of idle CPU, averaged per instance:

avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
)

rate() converts the ever-increasing counter into per-second usage; avg by (instance) collapses all CPU cores into one number per host. The result is a fraction between 0 and 1, where 0 means fully busy.

Add a Reduce expression (operation: Last) to turn the series into a single value per instance, then a Threshold expression with the condition IS BELOW 0.3 — fire when the host has less than 30% idle CPU. Set the evaluation behavior: evaluate every 1m, with a pending period of 5m so a brief spike doesn't page anyone. The condition must stay true for the full pending period before the alert fires.

Add labels such as severity=warning and team=ops — these drive routing in the next step. Before saving, click Preview (or Test Rule) to confirm the query returns the instances you expect.

Step 3: Route it to Slack via a contact point

A contact point is a reusable notification channel; a notification policy decides which alerts go to which contact point based on labels. Under Alerting → Contact points → New contact point, choose Slack and paste an incoming webhook URL for your workspace (created in Slack's app settings — treat the URL as a secret). Use the Test button here: Grafana sends a sample notification so you can verify the channel works before any real alert depends on it.

Then under Notification policies, add a nested policy matching the label team=ops, pointing at your Slack contact point. Set grouping to group by instance so one flapping host doesn't produce a dozen separate messages, and adjust group_wait (e.g., 30s) to batch alerts that fire together.

Trade-offs and limitations

• Evaluation load: every rule runs on the Grafana backend against your datasource. Hundreds of rules at 10-second intervals with heavy regex queries can measurably load both Grafana and Prometheus. Prefer 1-minute intervals unless you genuinely need faster detection.
• The 5-minute rate window plus a 5-minute pending period means roughly 10 minutes of latency before the first notification. For faster detection, shorten the rate window — but noisier data means more false positives.
• Single-point alerting: a single Grafana instance is a single point of failure for notifications. Grafana supports high availability for alerting (multiple instances coordinate via the database), but that is a separate setup step.
• Threshold semantics: avg by (instance) hides per-core hotspots. If a single saturated core matters (e.g., single-threaded workloads), alert on max by (instance, cpu) instead.

Verify before you trust it

Don't wait for a real outage. Create a temporary copy of the rule with a condition that is guaranteed true — for example, threshold IS BELOW 2, since idle CPU can never exceed 1 — and a short pending period. Confirm a notification arrives in Slack, then check Alerting → Notification history (or the logs) to see the evaluation and delivery record. Delete the canary rule afterward. That end-to-end test exercises the query, the threshold, the policy match, and the contact point — the four places this setup can silently break.