Centralizing Alerts with Grafana Unified Alerting: A Practical Guide

The problem: scattered alert rules and noisy notifications

Many teams inherit alerting setups where each data source has its own rule editor, legacy alerting UI, and scattered notification channels. When an incident occurs, engineers waste time hunting down the correct rule, adjusting thresholds in multiple places, and wading through duplicate or silenced notifications. This fragmentation makes it hard to version‑control alert logic and to enforce consistent routing policies.

Thesis: Grafana Unified Alerting puts rule creation, evaluation, and notification management in one place

Starting with Grafana 8.0, Unified Alerting replaces the legacy system with a single rule engine that can evaluate any supported data source, stores rules as JSON in the Grafana database, and centralizes notification policies and silencing. The result is a single source of truth for alerts that can be version‑controlled, reviewed, and deployed like any other configuration.

Section 1: Core concepts

Unified Alerting introduces three building blocks:

• Alert rule – a JSON object that defines a query, a condition (e.g., threshold > 80% for 5 m), and an optional for duration.
• Contact point – a reusable notification destination (Slack, email, webhook, etc.) that can be shared across rules.
• Notification policy – a tree that routes incoming alerts to contact points, adds timers (e.g., wait 10 m before escalating), and can silence alerts during maintenance windows.

Because rules are JSON, they can be stored in Git, reviewed via pull requests, and applied with the Grafana HTTP API or grafana-cli.

Section 2: Worked example – CPU usage alert on Prometheus

Assume a Prometheus data source named prometheus is already configured. The goal is to fire when CPU usage exceeds 80 % for five minutes and post to a Slack channel.

1. In the Grafana UI, go to Alerting → Rules → New alert rule. The unified builder opens.
2. Set Rule name to High CPU usage.
3. Under Data source, select prometheus.
4. In the Query panel, enter:

   avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[1m])) * 100
   

5. Click Add condition, choose When last() is above 80 for 5m.
6. Under Notifications, click Add contact point and select (or create) a Slack contact point that posts to #alerts.
7. Press Preview to see the rule evaluate against recent data. The preview shows whether the condition would be true or false; no notification is sent.
8. If the preview looks correct, click Save. The rule is now stored as JSON in the Grafana database.

The resulting rule JSON (visible via Alerting → Rules → (your rule) → View JSON) resembles:

{
  "title": "High CPU usage",
  "condition": {
    "type": "query",
    "evaluator": {
      "type": "gt",
      "params": [80]
    },
    "operator": {
      "type": "and"
    },
    "query": {
      "params": ["A"]
    },
    "reducer": {
      "type": "last"
    },
    "timeRange": "5m"
  },
  "data": [
    {
      "refId": "A",
      "relativeTimeRange": {
        "from": 0,
        "to": "now"
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "model": {
        "expr": "avg by (instance) (rate(node_cpu_seconds_total{mode!=\"idle\"}[1m])) * 100",
        "interval": "",
        "legendFormat": ""
      }
    }
  ],
  "notifications": [
    {
      "uid": "slack-alerts",
      "type": "slack"
    }
  ]
}

Section 3: Managing notifications with policies and silencing

Once contact points exist, you can build a notification policy:

• Go to Alerting → Notification policies.
• The default policy routes all alerts to the default contact point. Add a child policy that matches labels team=frontend and routes to a different Slack channel.
• Add a timer: wait 10m before sending the first notification, useful for flaky metrics.
• Create a silencing rule under Alerting → Silences to mute alerts during a planned maintenance window.

Because policies are evaluated hierarchically, you can keep a broad catch‑all rule while fine‑routing specific teams or services.

Trade‑offs and limitations

Unified Alerting aims to replace the legacy system, but a few gaps remain:

• Some complex cross‑query calculations (e.g., dividing two Prometheus queries that rely on different step sizes) are easier in the legacy editor; you may need to pre‑aggregate data in the query itself.
• Older alerting features like alert rule tags for legacy panel alerts are not available; you must rely on labels in the rule JSON.
• Teams accustomed to the point‑and‑click legacy UI may face a learning curve when editing JSON directly; investing in a shared schema or using the visual builder mitigates this.

To verify that your Grafana instance is ready, run these checks:

1. Confirm version: grafana-cli -v should return 8.0 or higher.
2. In the UI, navigate to Alerting → Rules and ensure the Create alert rule button opens the unified builder (not the legacy dialog).
3. Create a test rule using a simple query like up and press Preview; you should see an evaluation result without any notification being sent.

Actionable closing

Start small: migrate one non‑critical alert to Unified Alerting, store its JSON in a feature branch, and open a pull request for review. Use the preview function to validate the rule before merging. As confidence grows, bring more rules under the same system, adopt notification policies to reduce noise, and leverage silencing for planned work. By centralizing alert logic, you gain version‑controlled, reproducible alerts and clearer incident response—without the overhead of juggling multiple alerting UIs.