Grafana Alerting Architecture: Single-Instance to Multi-Cluster Design

The Problem: Alerting That Scales With Your Observability Stack

Teams adopting Grafana for metrics visualization eventually hit a wall: dashboards show problems, but nobody gets paged. The built-in alerting engine (introduced in Grafana 9.0) solves the basics—threshold evaluation, deduplication, multi-channel routing—but its single-instance design creates hard limits at ~5,000 rules. This note walks through the minimal viable architecture, where trust boundaries live, what to monitor, and the specific conditions that force a redesign toward Mimir, Loki, or a dedicated Alertmanager.

Requirements That Drive the Design

• Timely threshold evaluation: Rules must execute on a configurable schedule (default 1 minute) against Prometheus, Loki, or other data sources.
• Notification deduplication: Group alerts by labels so a single flapping series doesn't spam Slack.
• Multi-channel routing with severity: Route critical alerts to PagerDuty, warnings to email, info to webhook—each with independent silence windows.
• Silence and inhibition: Support ad-hoc silences (via UI or API) and inhibition rules (e.g., don't fire node-down if cluster-down is already firing).

Smallest Suitable Design: Single Instance, Built-In Engine

For most teams, a single Grafana instance with alerting enabled is the right starting point. The architecture looks like this:

• Grafana server runs the alert scheduler and evaluation loop in-process.
• Rule definitions live in the Grafana database (SQLite, MySQL, or Postgres) as provisioned resources or UI-created objects.
• Data source queries execute with the permissions of the user who created the rule—no elevated privileges.
• Notification channels store encrypted credentials (API keys, webhook URLs, SMTP passwords) using a key derived from grafana.ini's secret_key.

No external components required. Provision rules via file (/etc/grafana/provisioning/alerting/) or API (POST /api/v1/provisioning/alert-rules).

Trust and Data Boundaries

Understanding who can query what—and who can decrypt notification secrets—is the most overlooked part of alerting security.

Rule Evaluation Permissions

When a rule runs, the alert engine impersonates the rule creator. If user alice creates a rule querying Prometheus datasource prod-metrics, the evaluation uses alice's datasource permissions. Implication: if alice loses access to prod-metrics, the rule starts returning Error state. Audit by checking alert_rule_evaluation_errors_total per rule.

Notification Channel Secrets

Channel credentials are encrypted at rest using AES-256-GCM with a key derived from secret_key. Only the alert engine process can decrypt them at send time. Critical operational detail: rotating secret_key without re-saving each channel breaks all notifications until an admin edits and re-saves every channel in the UI or via API.

Data Source Isolation

The alert engine cannot query datasources the rule creator cannot access. There is no "superuser" evaluation mode. If you need rules that query restricted datasources, create a dedicated service account with the necessary permissions and provision rules as that user.

Operational Checks You Should Automate

Check	Metric / Method	Alert Threshold
Rule evaluation latency	alert_rule_evaluation_duration_seconds (histogram)	p99 > 30s for 5m
Rule evaluation errors	alert_rule_evaluation_errors_total (counter)	Any increase > 0
Notification send failures	alert_notifications_failed_total (counter)	Any increase > 0
Scheduler health	grafana_alerting_scheduler_tick_total (counter)	Stops incrementing
Rule sync status	GET /api/v1/provisioning/alert-rules → status: "OK"	Any rule not "OK"

Expose these via Prometheus scraping /metrics on the Grafana instance. Build a single "Grafana Alerting Health" dashboard with these five panels.

Failure Modes and Their Symptoms

1. Scheduler Stall

The in-process scheduler stops ticking. Rules never evaluate. Symptom: grafana_alerting_scheduler_tick_total flatlines; rule lastEvaluation timestamps grow stale in the UI. Cause: Go runtime GC pause, deadlock in custom datasource plugin, or OOM kill. Mitigation: Run Grafana with GOGC=50 and monitor process memory; alert on scheduler tick staleness.

2. Notification Channel Credential Expiry

Slack webhook URL rotates, SMTP password changes, PagerDuty integration key revoked. Symptom: alert_notifications_failed_total increments; notification log shows "401 Unauthorized" or "channel not found". Fix: Update channel in UI or via PUT /api/alert-notifications/:uid; test with the "Send test" button.

3. Data Source Unavailability

Prometheus endpoint returns 503, Loki query times out, CloudWatch throttles. Symptom: Rule state goes to Error; alert_rule_evaluation_errors_total increments; annotations show query error. Behavior: Alert stays in last known state (firing/resolved) until evaluation succeeds again—no automatic resolution.

4. Secret Key Rotation Without Re-encryption

Admin rotates secret_key in grafana.ini and restarts. All notification channels fail to decrypt. Symptom: Every notification attempt logs "decryption failed"; alert_notifications_failed_total spikes across all channels. Recovery: Re-save each channel (UI: edit → save; API: PUT with same payload). No automated fix exists.

When to Redesign: Concrete Triggers

Don't migrate prematurely. Move to a distributed alerting architecture when any of these conditions are true:

1. Rule count exceeds 4,000 (80% of the 5,000 default limit). Evaluation latency grows non-linearly; scheduler contention appears.
2. High availability required: Single Grafana instance is a SPOF. Active-passive with shared database works, but active-active needs external rule storage.
3. Cross-team rule isolation: Team A must not see Team B's rules or notification channels. Grafana's built-in RBAC (since 9.2) supports folder-level permissions, but evaluation still runs on the same process.
4. Data source access control stricter than Grafana RBAC: If security policy forbids the Grafana server process from holding credentials for certain datasources, you need a sidecar evaluator (e.g., Mimir ruler) that holds only the necessary tokens.
5. Unified alert routing across Grafana + non-Grafana sources: Kubernetes events, Cortex/Thanos ruler alerts, and Grafana rules all need to hit the same Alertmanager for deduplication and inhibition.

Redesign Target: External Rule Storage + Alertmanager

The standard path: migrate rule definitions to Grafana Mimir or Loki rule groups (YAML files stored in object storage or Git), run a dedicated ruler component, and ship alerts to a centralized Alertmanager cluster. Grafana instances become read-only dashboards for alert state.

Migration steps (high level):

1. Export existing rules: GET /api/v1/provisioning/alert-rules → convert to Mimir rule group YAML.
2. Deploy Mimir ruler with -ruler.rule-path pointing to the YAML directory.
3. Configure ruler -ruler.alertmanager-url to your Alertmanager cluster.
4. Update Grafana datasources to point at Mimir/Alertmanager for alert state queries.
5. Disable built-in alerting in Grafana: alerting.enabled = false in grafana.ini.

Verification Checklist Before Going Live

Run these in a staging environment that mirrors production datasource topology:

1. Provision a test rule via API:

   curl -X POST -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"title":"Test CPU High","condition":"A","data":[{"refId":"A","relativeTimeRange":{"from":300,"to":0},"datasourceUid":"prom-prod","model":{"expr":"avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) by (instance) < 0.1","intervalMs":1000}}],"intervalSeconds":60,"annotations":{"summary":"High CPU"},"labels":{"severity":"warning"}}' \
     https://grafana.example.com/api/v1/provisioning/alert-rules

   Verify rule appears in UI with status OK.
2. Test notification end-to-end: In UI → Alerting → Contact points → Edit your Slack/email channel → "Send test". Confirm receipt. Then check alert_notifications_sent_total increased on /metrics.
3. Simulate datasource failure: Revoke the API key for the Prometheus datasource used by the test rule. Wait one evaluation cycle. Verify rule state shows Error and alert_rule_evaluation_errors_total incremented.
4. Validate secret key rotation procedure: Change secret_key, restart Grafana, attempt test notification. Observe failure. Re-save channel. Confirm recovery. Document this runbook.

Limitations to Accept in the Single-Instance Design

• No horizontal scaling of evaluation—single scheduler, single process.
• Rule limit ~5,000 (configurable via alerting.max_alert_rules but not recommended higher).
• No cross-instance deduplication—two Grafana replicas evaluating the same rule will send duplicate notifications unless you use external Alertmanager.
• Notification channel secrets tied to Grafana's secret_key; no external KMS integration.

Practical Way to Check Current State

Run this query against your Grafana's Prometheus scrape target:

sum by (instance) (grafana_alerting_active_alerts) > 0
  and
sum by (instance) (rate(grafana_alerting_scheduler_tick_total[5m])) == 0

If any instance returns true, that Grafana's scheduler has stalled. Alert immediately.