Grafana Unified Alerting: Architecture Decisions for Production Deployments

The Problem: Centralized Alerting Without Operational Surprises

Teams adopting Grafana Unified Alerting (Grafana 8.0+) often start with a simple goal: write alert rules in the UI or Git, have Grafana evaluate them server-side, and route notifications through a single pipeline. The hidden complexity appears at scale — rule evaluation storms OOM the Grafana container, Alertmanager downtime silently drops notifications, and RBAC gaps let developers modify routing policies they shouldn't touch. This note maps the architecture boundaries so you can size the deployment correctly from day one.

Requirements That Drive the Design

• Centralized authoring: Rules defined in Grafana UI or provisioned via YAML/JSON (ConfigMap, Helm, GitOps), not scattered across Prometheus rule files.
• Server-side evaluation: The embedded alerting engine runs in the Grafana process, querying data sources (Prometheus, Loki, Mimir, CloudWatch) on a schedule. Data source credentials never leave the Grafana server.
• Multi-tenant isolation: Each tenant sees only their rules, folders, and notification policies. Evaluation runs in the same process but scoped by organization ID.
• Configurable routing: Notification policies map rule labels to receivers (PagerDuty, Slack, Opsgenie, webhook) with grouping, inhibition, and silencing — handled by Alertmanager (internal or external).

Smallest Suitable Production Design

For a single team or small organization, start here:

• One Grafana instance with alerting.enabled=true (default since 9.0). The embedded alerting engine evaluates rules on the configured interval.
• One external Alertmanager instance (not the embedded one — see failure modes). Deploy via Helm chart prometheus-community/alertmanager with persistence enabled. Grafana connects via alerting.alertmanager_url in grafana.ini or environment variable GF_ALERTING_ALERTMANAGER_URL.
• Provisioned rules: Store rule definitions in Git as YAML with stable uid fields. Apply via ConfigMap or Grafana's HTTP API during CI/CD. Example rule file:

apiVersion: 1
groups:
  - name: instance-down
    folder: Infrastructure
    interval: 60s
    rules:
      - uid: instance-down-01
        title: Instance Down
        condition: B
        data:
          - refId: A
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus-prod
            model:
              expr: up == 0
              intervalMs: 60000
              refId: A
          - refId: B
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params: [1]
                    type: gt
                  operator:
                    type: and
                  query:
                    params: [A]
                  reducer:
                    params: []
                    type: last
                  type: query
              refId: B
        annotations:
          summary: 'Instance {{ $labels.instance }} down for 5m'
        labels:
          severity: critical
          team: platform

• Notification policies: Configured in Grafana UI (Alerting → Notification policies) or provisioned via API. Map severity=critical to PagerDuty receiver, severity=warning to Slack. Set group_by: ['alertname', 'cluster'] and group_wait: 30s to avoid notification storms.

Trust and Data Boundaries

Boundary	Who Accesses	What Crosses	Risk If Misconfigured
Alert rule authoring	Editor role on folder	Rule definition (query, labels, annotations)	Developer creates rule querying sensitive data source they shouldn't read
Notification policy admin	Admin role (global)	Routing tree, receiver configs, inhibition rules	Team routes critical alerts to dev Slack instead of PagerDuty
Alertmanager config	Platform team (infra repo)	Receiver secrets (API keys), Go templates, HA gossip mesh	Template syntax error breaks all notifications silently
Rule evaluation	Grafana server process	Query results → labels/annotations only (no raw series)	CPU spike during evaluation storm OOMs container

Key invariant: Data source credentials stay in Grafana. Only labels and annotations (the alert payload) reach Alertmanager. Tenant data never crosses organization boundaries in the evaluation engine.

Operational Checks You Can Run Today

Rule Health

Run from any host with network access to Grafana (requires API key or basic auth with Viewer role):

curl -H "Authorization: Bearer $GRAFANA_API_KEY" \
  https://grafana.example.com/api/alerts/rules | jq '.[] | {uid, title, state, lastEvaluation, evaluationError}'

Check: state should be "Normal" or "Pending". evaluationError non-null means the rule failed to evaluate (timeout, query error). lastEvaluation older than 2× interval indicates scheduler lag.

Alertmanager Queue Health

Run from host with access to Alertmanager (port 9093):

curl -s http://alertmanager:9093/metrics | grep -E 'alertmanager_notifications_|alertmanager_queue_'

Watch: alertmanager_queue_length growing = notifications backing up. alertmanager_notifications_failed_total increasing = receiver errors (bad webhook URL, auth failure).

UI Verification

In Grafana: Alerting → Alert rules. Confirm each rule shows a green "Normal" badge and a recent "Last evaluated" timestamp. Rules in "Error" state with "evaluation timeout" need interval increase or query optimization.

Failure Modes and Mitigations

Failure	Symptom	Root Cause	Mitigation
Rule evaluation timeout	Rule state "Error", evaluationError: "timeout"	Query exceeds 30s default (alerting.evaluation_timeout)	Increase timeout or optimize query; set interval >= 60s for heavy queries
Alertmanager unavailable	Alerts fire in Grafana UI but no notifications sent	Embedded Alertmanager (Grafana <10) has no persistence; external AM down	Always use external Alertmanager with PVC; Grafana queues in-memory only
Notification storm	Hundreds of Slack/PagerDuty alerts in minutes	Misconfigured group_by or missing inhibition rules	Set group_by: ['alertname', 'cluster'], add inhibition: critical inhibits warning
Provisioning conflict	Rules flip-flop between versions on reload	Duplicate uid across files; last-write-wins	Enforce uid generation in CI: uid: {{ .RuleName }}-{{ hash .Expr }}
Grafana OOM during evaluation	Container restart, rules show "Error" after restart	Too many concurrent evaluations; no resource limits	Set resources.limits.memory=2Gi, alerting.concurrent_render_limit=5, interval >= 60s

Conditions That Change the Design

>5,000 Active Rules

Migrate rule evaluation to Grafana Mimir Ruler or a dedicated alerting cluster. Grafana becomes read-only for UI; rules provisioned to Mimir via mimirtool or Helm. Grafana connects to Mimir as a data source for alert state. This separates evaluation CPU from the UI/API process.

Multi-Region HA

Run an external Alertmanager cluster (3+ nodes) with gossip mesh (--cluster.peer). Grafana replicas in each region point to local Alertmanager. Grafana UI replicas run read-only (alerting.enabled=false) — only one region evaluates rules (active/passive) or use Mimir Ruler for active/active.

Strict Compliance (Change Control)

Disable UI editing entirely: alerting.disable_ui_editing=true. All rules and notification policies provisioned via GitOps with PR review gates. Use grafana-operator or Terraform to enforce. Auditors get immutable audit trail in Git.

Cost-Sensitive / Small Scale

Use built-in receivers (Grafana 10+): Slack, PagerDuty, Opsgenie, webhook configured directly in Grafana notification policies — no external Alertmanager. Trade-off: no deduplication across Grafana replicas, no persistence if Grafana restarts. Acceptable for <500 rules, single replica, non-critical paths.

Verification Checklist Before Go-Live

1. Deploy Grafana 10.x with alerting.enabled=true and external Alertmanager URL configured.
2. Provision a test rule via ConfigMap (use the example above). Verify it appears in UI with "Normal" state and a recent evaluation timestamp.
3. Trigger the rule condition (e.g., stop a test target). Confirm Alertmanager receives it: curl http://alertmanager:9093/api/v2/alerts | jq.
4. Verify notification delivers to test receiver (use webhook.site or Slack test channel).
5. Simulate Alertmanager downtime: stop Alertmanager, fire alert, restart Grafana. Confirm alert state persists in Grafana UI (stored in Grafana DB) but notification is lost — proves need for external AM with persistence.
6. Load test: create 1,000 rules evaluating every 30s against Prometheus. Monitor grafana_alerting_rule_evaluation_duration_seconds histogram and container CPU/memory.
7. Validate RBAC: create user with Viewer role on folder — confirm cannot edit rules. Grant Editor — confirm can create/edit rules but not notification policies (requires Admin).

Limitations to Track

• Embedded Alertmanager (Grafana <10): No HA, no persistence. Not production-suitable. Upgrade to 10+ or deploy external.
• No per-rule RBAC: Permissions are folder-scoped. Plan folder hierarchy (e.g., per-team folders) before onboarding.
• Template errors surface at send time: Alertmanager Go templates only validate on notification send. Test with amtool test in CI: amtool test config.yml --rule-file=rules.yml.
• Duplicate UID non-determinism: Grafana loads last-write-wins. Enforce UID generation in CI pipeline.

Quick Diagnostic Decision Tree

Alert not firing?
├─ Rule state "Error" → Check evaluationError, fix query
├─ Rule state "Normal" but condition met → Verify interval
├─ Rule state "Firing" but no notification → Check AM API
└─ Notification sent but not received → Test receiver endpoint