Configuring Grafana 10+ Alerts with Notification Channels: A Practical Guide

Why You Should Use Grafana’s Unified Alerting

Grafana 10 introduced a single alerting engine that replaces the legacy Alertmanager. All alerts are evaluated on the Grafana server, and notifications are sent through notification channels that can be reused across rules. This simplifies management and gives you tighter control over delivery.

Quick Takeaway

To set up an alert that notifies a Slack channel when a Prometheus query exceeds a threshold, you need to:

1. Create a Slack notification channel with the webhook URL.
2. Define an alert rule in a dashboard panel or via the API, referencing the channel in the annotations.
3. Verify the rule evaluates and the notification is sent.

Step 1 – Create a Notification Channel

Navigate to Settings → Notification channels and click Add channel. Choose the channel type (e.g., Slack) and fill in the required fields:

• Name: A human‑readable identifier, e.g., Slack‑Ops.
• Type: Slack.
• URL: The incoming webhook URL from your Slack workspace.
• Message template (optional): Use Grafana templating such as ${__value.avg} or ${__time} to inject context.

After saving, the channel is available for all alert rules. No Grafana restart is required for this change.

Step 2 – Define an Alert Rule

In Grafana 10+, you can create an alert directly from a panel or via the API. Below is a panel‑based example that uses a Prometheus query.

# Panel query (Prometheus)
rate(http_requests_total[5m]) > 100

When editing the panel, open the Alert tab and click Create rule. Configure the rule as follows:

• Name: High HTTP Request Rate
• Expression: The Prometheus query above.
• For: 5m – ensures the condition holds for 5 minutes before firing.
• Annotations: Add a key notification_channel with value Slack‑Ops (the channel name you created).
• Message: HTTP request rate exceeded 100 requests/second. Avg: ${__value.avg}

Save the panel. The alert engine will evaluate the rule every eval_interval (default 1 minute). You can see the rule’s status in the Alert tab of the dashboard.

API‑Based Creation (Optional)

For automation, POST to /api/alerts with JSON:

{
  "name": "High HTTP Request Rate",
  "datasourceUid": "",
  "condition": {
    "type": "query",
    "query": {
      "refId": "A",
      "datasourceUid": "",
      "model": {
        "queryType": "prometheus",
        "expr": "rate(http_requests_total[5m]) > 100"
      }
    },
    "for": "5m"
  },
  "annotations": {
    "notification_channel": "Slack‑Ops"
  },
  "message": "HTTP request rate exceeded 100 requests/second. Avg: ${__value.avg}"
}

Replace <UID> with the actual datasource UID. A 200 OK response indicates success.

Common Pitfalls and How to Avoid Them

• Missing for duration: Without a for clause, the alert fires immediately on the first evaluation that meets the condition, leading to flapping. Always set a reasonable for value.
• Wrong channel name in annotations: The key must match exactly the channel’s name. A typo results in the alert not being routed.
• Query returns no data: If the Prometheus query is syntactically correct but returns no series, Grafana treats it as a non‑trigger and the alert never fires. Test the query in the Explore panel first.
• Authentication token errors: For Slack, ensure the webhook URL is correct and the workspace allows incoming webhooks. A 403 or 404 response in the Grafana logs indicates a bad token.
• Exceeding the evaluation limit: Grafana caps concurrent evaluations per datasource to 10 by default. If you have many alerts on a single Prometheus datasource, you may experience delays. Adjust maxConcurrentEvaluations in grafana.ini if needed.

Version‑Sensitive Considerations

Grafana 10+ uses the unified alerting API. In earlier 9.x releases, you would configure Alertmanager separately. If you upgrade, remember:

• Alert rules defined in dashboard.json are migrated automatically, but older alertmanager.yml files are ignored.
• Notification channel IDs are now strings; older numeric IDs may cause failures.
• Some legacy annotations (e.g., alertmanager) are deprecated.

How to Verify the Setup

1. Trigger the condition manually (e.g., set the query to 1 > 0) and observe the alert state change to Alerting in the UI.
2. Check the Slack channel for the notification. The message should contain the templated values.
3. Inspect Grafana logs for entries like Alert rule evaluated and Notification sent to confirm the engine processed the rule.
4. Use curl -H "Authorization: Bearer <token>" https://<grafana-host>/api/alerts to list active alerts and verify the rule appears with status active.

Limitations and What to Watch For

• Alert evaluation happens on the Grafana server; if the server is overloaded or offline, alerts may miss triggers.
• External services like Slack have rate limits. Rapidly firing alerts can exhaust these limits, causing subsequent notifications to be dropped.
• Prometheus query errors are only caught at evaluation time, so a typo may silently silence an alert until the next evaluation.
• Changing a notification channel’s URL or token requires no restart, but if you change the channel’s name, you must update all alert annotations that reference it.

Final Checklist Before Going Live

• Verify the notification channel works by sending a test message.
• Confirm the alert rule’s for duration and expression are correct.
• Monitor Grafana logs for any evaluation or notification errors.
• Consider setting up Grafana's built‑in alert history to review past alerts.