Tuning Grafana Alert Rule Evaluation Intervals for Cost‑Effective Monitoring

Problem: Alerts arrive too late or cost too much

Teams often notice two opposing symptoms in their Grafana‑based monitoring:

• Critical spikes are missed because the alert rule only runs once per minute.
• Prometheus or Loki instances show high query rates, leading to increased infrastructure cost or throttling.

The root cause is the same setting: the evaluation_interval of each alert rule. This interval controls how often Grafana re‑evaluates the rule against its datasource.

Thesis: Align evaluation frequency with metric granularity and service SLA

By tuning evaluation_interval to match the resolution of the underlying metrics and the required alerting latency, you can reduce unnecessary queries while still meeting your alerting SLA.

Understanding evaluation_interval

Grafana alert rules are evaluated at a configurable interval defined by the evaluation_interval field in the rule definition. For most datasources (Prometheus, Loki, MySQL, etc.) the default is 1m. The interval is independent of the datasource’s scrape interval; it tells Grafana how often to issue a query for that rule.

When the interval is shortened:

• Detection latency improves (alerts can fire sooner after a condition appears).
• Query load on the datasource increases proportionally.

When the interval is lengthened:

• Query load drops, reducing cost and the risk of hitting remote‑datasource rate limits.
• There is a chance that short‑lived spikes occur and disappear between two evaluations, causing missed alerts.

Worked example: Tuning a Prometheus‑based rule

Assume you have a Prometheus datasource scraped every 15s and an alert rule that fires when the 5‑minute rate of HTTP errors exceeds 0.01. The rule currently uses the default 1m evaluation interval.

Goal: Reduce query load by half while still detecting errors that persist for at least 30s.

1. Determine the minimum useful interval. Since the metric is available every 15s and you need to catch a condition lasting 30s, an evaluation every 30s guarantees at least two samples during a qualifying event.
2. Update the rule via the Grafana API (requires Admin or Editor permissions).

   # PATCH /api/v1/provisioning/alert-rules/
   {
     "data": {
       "title": "High HTTP error rate",
       "condition": {
         "type": "query",
         "evaluator": {
           "type": "gt",
           "params": [0.01]
         },
         "operator": {
           "type": "and"
         },
         "query": {
           "params": ["A", "5m", ""]
         }
       },
       "executionErrorState": "alerting",
       "for": "0m",
       "interval": "30s",   # <-- changed from 1m to 30s
       "datasourceUid": "prometheus-prod",
       "model": {
         "conditions": [
           {
             "evaluator": {
               "params": [0.01],
               "type": "gt"
             },
             "operator": {
               "type": "and"
             },
             "query": {
               "params": ["A", "5m", ""]
             },
             "reducer": {
               "params": [],
               "type": "last"
             },
             "type": "query"
           }
         ],
         "datasource": {
           "type": "prometheus",
           "uid": "prometheus-prod"
         },
         "interval": "30s",
         "title": "High HTTP error rate",
         "version": 1
       }
     }
   }

   Run this command from a machine with network access to your Grafana instance, using an API key with Admin scope.

3. Verify the change.
   • In the Grafana UI, go to Alerting → Rule list and locate the rule. The Evaluation interval column should now show 30s.
   • Alternatively, call GET /api/v1/provisioning/alert-rules/ and check the interval field in the JSON response.
4. Check the impact on query load.

   Grafana exposes internal metrics via grafana_exporter. Look for the counter grafana_alert_rule_evaluations_total split by rule UID. After the change, the rate should be roughly half of the previous value (assuming the rule was previously evaluated every minute).

   On the Prometheus side, monitor prometheus_http_requests_total for the endpoint that serves the rule’s query. You should see a corresponding drop in request frequency.

Trade‑off and limitation

The primary limitation of lengthening evaluation_interval is the risk of missing transient events that are shorter than the interval. In the example, a spike that lasts only 20s could be invisible to the rule because the two evaluation points might fall before and after the spike. To mitigate this:

• Use recording rules in Prometheus to pre‑aggregate metrics at a finer granularity (e.g., a 10‑second moving average) and alert on the recording rule.
• Set a separate, shorter‑interval rule only for the most critical, short‑duration symptoms, accepting the higher cost for those specific metrics.

Another operational risk is hitting the datasource’s query concurrency limit. If you shorten intervals across many rules simultaneously, verify that the datasource can handle the increased load (check Prometheus prometheus_engine_query_duration_seconds or Loki loglines_processed_total).

Actionable closing

Start by inventorying your alert rules and noting their current evaluation_interval. For each rule, ask:

1. What is the native scrape/resolution interval of the underlying metric?
2. What is the minimum duration of a condition that must trigger an alert?
3. Does the current interval align with those two values?

Adjust the interval to the smallest value that satisfies both constraints, then verify via the UI or API and monitor query load for a few minutes. Document the change and set a reminder to re‑evaluate after any modification to the datasource scrape schedule or after adding new rules that share the same datasource.

By treating evaluation_interval as a tunable parameter rather than a fixed default, you can achieve measurable cost savings without sacrificing the timeliness of your alerts.