Prometheus Recording Rules: Precompute the Queries Your Dashboards Keep Re-Running

If a Grafana dashboard takes ten seconds to load, the usual culprit is not your network or Grafana itself — it is a PromQL expression like sum(rate(http_requests_total[5m])) by (job) being re-evaluated against raw samples every time someone refreshes a panel. Recording rules are Prometheus's built-in answer: they evaluate an expensive expression on a schedule and store the result as a brand-new time series, so dashboards and alerts query one cheap series instead of re-scanning raw data.

The thesis is simple: raw metrics stay untouched, and you pay the query cost once per evaluation interval instead of once per viewer. The trade-off is that every recorded expression becomes a persisted series in your local TSDB, so this is a tool to apply deliberately, not preemptively.

What a recording rule actually does

A recording rule is a PromQL expression plus a name. Prometheus evaluates it at a configurable interval and writes the result as a new series under that name, stored in the TSDB exactly like a scraped metric. That means recorded series can be graphed, alerted on, federated, and even fed into other recording rules.

Rules live in rule files loaded via rule_files in prometheus.yml, and they run in the same engine as alerting rules. Rules within a group execute sequentially at the group's interval, so putting related derived metrics in one group keeps them consistent with each other — they are computed from the same evaluation timestamp.

A worked example: precomputing a request rate

Suppose your dashboards and an alert both need the per-job HTTP request rate over five minutes. Instead of repeating the raw aggregation everywhere, create a rule file, say /etc/prometheus/rules/http.rules.yml:

groups:
  - name: http_request_rates
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)
      - record: job:http_errors:ratio5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
            /
          sum(rate(http_requests_total[5m])) by (job)

Then reference it from the Prometheus config (this requires a config reload — send SIGHUP to the Prometheus process, POST to /-/reload if the lifecycle API is enabled, or restart the service; all require appropriate host permissions):

rule_files:
  - /etc/prometheus/rules/*.rules.yml

Now the dashboard panel queries job:http_requests:rate5m — a single series per job — instead of scanning five minutes of raw samples across every instance. The alert expression uses job:http_errors:ratio5m and evaluates in milliseconds.

Two details in that example are deliberate. First, the naming convention level:metric:operation (here job:http_requests:rate5m) makes precomputed series discoverable and prevents collisions with raw metric names. Second, the group interval is set to 30s to match a typical scrape_interval. Aligning the two means each evaluation sees fresh samples, so derived series do not look stale relative to the raw data.

Verify before and after deploying

Before loading the file, validate it with promtool check rules /etc/prometheus/rules/http.rules.yml, run on the Prometheus host as a user with read access to the rule files. This catches syntax and expression errors that would otherwise surface only as a silently missing series.

After reloading, confirm three things:

• Open the /rules page (or query /api/v1/rules) and check the group is loaded, healthy, and evaluating on schedule.
• In the expression browser, query the recorded name — it should appear after one evaluation interval. If it does not, the group health field usually tells you why.
• Compare latency: time the raw aggregation against the recorded series using Grafana's query inspector or Prometheus's query log. This gives you the actual justification for keeping the rule.

The trade-off: recorded series are real series

Because recorded results live in the TSDB like any other metric, they consume storage and count toward your series cardinality. Recording an aggregation that preserves a high-cardinality label — say, by (job, instance, path) where path has thousands of values — can cost more than querying the raw data ever did. Record at the coarsest label set your consumers actually need.

Two more limitations worth planning around:

• No history. A recording rule only produces data from the moment it is deployed. Dashboards will show a gap before that point. Recent Prometheus versions ship a promtool backfill feature (promtool tsdb create-blocks-from rules) that can retroactively materialize recorded series from existing data, but the exact flags are version-sensitive — check promtool --help on your installed version before relying on it.
• Silent absence. A rule with a typo or an expression that matches nothing yields an absent series, not an error on your dashboard. For critical derived metrics, consider an alert on missing data, e.g. using absent(job:http_errors:ratio5m).

When to add one — and when not to

The actionable rule of thumb: add a recording rule when the same aggregation is evaluated repeatedly — by a dashboard with many viewers, by multiple panels, or by an alert over a long range — and you can measure that it is slow. Skip it for ad-hoc exploration, one-off queries, or expressions that are already cheap. Over-recording is a real failure mode; every rule should be justified by an actual query cost you have observed, then verified after deployment with the latency comparison above. Start with your slowest dashboard panel, record that one expression, and measure the difference.