Stop Making Grafana Do the Math: Prometheus Recording Rules in Practice

Your dashboard takes 20 seconds to load. The culprit is usually the same: a handful of panels each running a heavy PromQL expression — a sum by (...) over a rate(...[5m]) across thousands of raw series — and every refresh recomputes all of it from scratch. Alerts do the same thing every evaluation cycle. Recording rules exist to fix exactly this: Prometheus evaluates the expensive expression once, on a schedule, and stores the result as a brand-new time series that everything else queries cheaply.

The thesis is simple: if an expression is expensive and queried more than once, precompute it. If it's cheap or queried rarely, leave it alone — recording rules are not free.

What a recording rule actually does

A recording rule is a PromQL expression that Prometheus evaluates at a fixed interval and writes back into its own storage as a new metric. Rules live in YAML files organized into groups; all rules in one group run sequentially at that group's interval, while groups run independently of each other. That last detail matters when one rule consumes another rule's output — they need to be in the same group, evaluated in order, or you accept a full interval of lag between them.

You load rule files via rule_files in prometheus.yml:

rule_files:
  - /etc/prometheus/rules/*.yml

Edit the file, then either reload Prometheus (kill -HUP <pid> or POST to /-/reload if the lifecycle API is enabled) or restart it. Before doing either, validate the file — run this on the Prometheus host (or anywhere with promtool installed), no special permissions needed beyond reading the file:

promtool check rules /etc/prometheus/rules/http.yml

A syntax error in a rule file can prevent a clean reload or silently drop the group, so this check is worth scripting into CI.

A worked example: per-job request rate

The classic case. Your dashboards and alerts repeatedly compute total request rate per job:

sum by (job) (rate(http_requests_total[5m]))

On a busy server this scans five minutes of every http_requests_total series on every evaluation. Instead, record it once:

groups:
  - name: http_aggregates
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

Now panels and alerts query job:http_requests:rate5m — a single precomputed series per job — and the expensive range-vector evaluation happens once per interval instead of per viewer. Because alerting rules share the same file format, you can colocate the alert that consumes this series in the same file, which keeps the logic reviewable in one place.

Note the naming convention level:metric:operations. It tells the next engineer, at a glance, the aggregation level (job), the source metric, and the operation plus window (rate5m). Adopt it from day one; unnamed recorded metrics become archaeology within months.

Verifying it works

After one evaluation interval, query the new series in the Prometheus expression browser or via /api/v1/query. If nothing appears, check the server's rule-evaluation metrics: prometheus_rule_evaluations_total should be incrementing for your group, and rule evaluation duration metrics tell you whether the group finishes within its interval. A group whose evaluation time exceeds its interval will fall behind — that's a signal to split the group or raise the interval.

To confirm the win, time the old raw expression and the new recorded query in the UI and compare. The recorded query should drop from seconds to milliseconds on large instances.

The trade-offs nobody mentions first

Recording rules are not a universal speedup, and they carry real costs:

• Storage and ingestion. Every recorded series consumes disk and memory for the entire retention period. Recording sum by (pod) (...) in a churny Kubernetes environment can create more cardinality than the original query cost you. Aggregate away high-cardinality labels before recording, not after.
• Freshness lag. A rule on a 30s interval can lag reality by up to the interval plus evaluation time. For fast-burn alerts where seconds matter, a recorded input may be too stale — evaluate the raw expression in the alert instead.
• No backfill. Changing a rule's expression only affects data from that moment forward. Historical panels mixing old and new logic will silently disagree with each other.
• No data repair. Rules evaluate against whatever the local server scraped. Scrape gaps become gaps in the recorded series too.

Exact syntax details and defaults vary across Prometheus versions, so check the docs for the version you actually run before copying anything into production.

Where to start

Open your slowest dashboard, find the expression evaluated most often, and record it using the naming convention above. Validate with promtool check rules, reload, confirm the series appears after one interval, then point the dashboard at the recorded metric and measure the difference. One rule, one measurable win — then decide whether the pattern earns a broader rollout.