Choosing Grafana Unified Alerting vs Legacy Alerting for Production Pipelines

Problem & Takeaway

Grafana 8.0 introduced a new alerting engine that moves rule evaluation out of the Grafana server process. Production teams that rely on alerts must decide whether to keep the legacy in‑process system or migrate to the Unified engine. The decision hinges on architecture, high‑availability (HA) requirements, data‑source coverage, and upgrade timelines.

Decision Constraints

• Grafana version – Unified Alerting requires 8.0+. Legacy is deprecated and UI‑removed in 11.0.
• High‑availability – Unified supports HA via a shared rule store or external Alertmanager; Legacy duplicates notifications across replicas.
• Data‑source support – Unified natively alerts on Prometheus, Cortex, Loki, Tempo, and Mimir. Legacy only supports Prometheus and Graphite.
• Rule count & performance – Unified scales to tens of thousands of rules with modest RAM; Legacy struggles beyond ~1,000 rules.
• Operational familiarity – Unified uses Go templates and a separate provisioning model; Legacy is tied to dashboard panels.
• Upgrade path – If you plan to move past Grafana 10.x, you must migrate before 11.0, otherwise alerts become read‑only.

Feature Comparison Table

Feature	Unified Alerting (8.0+)	Legacy Alerting (pre‑8.0)	Notes
Evaluation engine	Dedicated Alerting engine (embedded or external)	In‑process Grafana server loop	Unified reduces server load
HA support	Shared rule store + Alertmanager routing	Independent evaluation per replica → duplicate alerts	Unified preferable for HA
Data source coverage	Prometheus, Cortex, Loki, Tempo, Mimir	Prometheus, Graphite only	Unified needed for log/trace alerts
Rule provisioning	YAML/JSON files or HTTP API	Only via dashboard JSON (alert.list)	Unified allows GitOps
Notification templating	Go templates with full label set	Limited context (value, metric, dashboard UID)	Unified offers richer payloads
Silence & inhibition	Label‑based rules, external Alertmanager integration	None – silences tied to panel	Unified required for complex silences
Resource footprint	~50‑200 MB RAM per 10k active rules	Negligible but scales poorly beyond 1k rules	Unified scales better
Migration tooling	CLI: grafana cli admin alerts migrate and validate-rule-file	No built‑in migration path	Unified simplifies migration

Trade‑off Analysis

• Evaluation location – Moving evaluation to the Alerting engine decouples it from Grafana’s web stack, improving UI responsiveness and isolating failures.
• HA & deduplication – Unified’s integration with Alertmanager ensures a single notification per alert across replicas. Legacy’s per‑instance evaluation leads to split‑brain and duplicate alerts unless custom deduplication logic is added.
• Feature richness – Unified supports multi‑dimensional labels, inhibition rules, and external routing. Legacy lacks these, making complex alert logic hard to express.
• Operational overhead – Unified introduces a new component to monitor. The additional RAM usage is modest but requires capacity planning. Legacy adds virtually no overhead.
• Upgrade risk – Legacy alerts become read‑only in Grafana 11.0. Teams that haven’t migrated risk losing alerting functionality after the upgrade.

Concrete Migration Example

Below is a step‑by‑step example that converts a legacy dashboard panel alert into a Unified rule, validates it, and optionally points it to an external Alertmanager.

1. Export the legacy alert definition

# Run on the Grafana host
curl -s -u admin:admin \
  http://localhost:3000/api/dashboards/uid/abcd1234 \
  | jq '.dashboard.panels[] | select(.alert) | {id, title, alert}' > legacy_alert.json

Replace abcd1234 with your dashboard UID. The JSON contains the panel alert configuration.

2. Convert to Unified rule YAML

# Example conversion for a Prometheus rule
apiVersion: 1
groups:
- name: example-group
  rules:
  - alert: HighCPUUsage
    expr: sum(rate(container_cpu_usage_seconds_total{job="node_exporter"}[5m])) > 0.9
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "CPU usage above 90%"
      description: "CPU usage on {{ $labels.instance }} has been above 90% for 5 minutes."

Place this YAML in /etc/grafana/alert_rules.yaml (or a shared GitOps repo). The apiVersion and groups structure matches the file provider spec.

3. Validate the rule file

grafana cli admin alerts validate-rule-file -f /etc/grafana/alert_rules.yaml

Expected output: ✓ Validation successful. If there are template errors or unsupported labels, the CLI will report them.

4. Dry‑run the rule against live data

grafana cli admin alerts test-rule -f /etc/grafana/alert_rules.yaml --dry-run

This command evaluates the rule against the current Prometheus data and prints the number of firing instances without sending notifications.

5. Optional: Point to an external Alertmanager

In grafana.ini add:

[alerting]
alertmanager_url = http://alertmanager:9093

Ensure the Alertmanager route config matches the labels used in the rule (e.g., severity: warning) to avoid dropped alerts. Verify routing with:

amtool check-config
amtool config routes.test

6. Verify HA deduplication (if running multiple replicas)

Trigger the same alert on two Grafana instances and confirm a single notification with count=2 in the Alertmanager UI.

Validation Checklist

• Grafana version ≥ 8.0 and grafana-cli -v shows the engine enabled.
• Legacy alert endpoints (GET /api/alerts) return 404, confirming Unified is active.
• Rule provisioning file is present and readable by Grafana.
• Alert evaluation metrics (e.g., evaluation_duration_seconds) remain within acceptable bounds (<2 s per rule).
• External Alertmanager receives and deduplicates alerts; no silent suppression due to mis‑configured routes.

Limitations & Caveats

• The migration CLI is available from Grafana 9.0+. On 8.x you must manually convert rules.
• Label mismatches between Unified rules and Alertmanager routes can silently drop alerts; double‑check the matchers section.
• Unified’s for duration starts when the condition first fires, adding up to one evaluation interval of latency compared to Legacy.
• HA with embedded engine requires a shared rule store (e.g., NFS, GitOps). Divergent rule sets lead to split‑brain alert states.
• Loki/Tempo alert queries (LogQL/TraceQL) may have higher latency; test at production load before setting intervals below 30 s.

Conclusion

For production environments that need HA, multi‑source alerts, or plan to upgrade beyond Grafana 10.x, Unified Alerting is the safer, feature‑rich choice. Legacy Alerting remains viable only for small, single‑instance setups that exclusively use Prometheus or Graphite and can tolerate duplicate notifications.