Reduce Prometheus dashboard load with recording rules

Useful answer

If your dashboards slow down because every refresh recomputes rate(http_requests_total[5m]) across thousands of raw samples, Prometheus recording rules can help. A recording rule evaluates a PromQL expression on a fixed schedule and writes the result as a new time series. Your dashboard then queries that pre-aggregated series instead of scanning raw data on demand.

This guide shows one worked configuration, explains where the load moves, and lists the cardinality and reset mistakes that make recording rules more expensive than the queries they replace.

How recording rules work

Assumes Prometheus 2.x with YAML rule files. A recording rule lives in a rule group. Prometheus evaluates the group’s expressions at the group interval, which inherits the global scrape interval unless overridden. Each successful evaluation creates samples for a new metric named with the record: directive. You can reshape labels with the optional labels: clause, which is also the main lever for controlling output cardinality.

The computation moves from query time to evaluation time. Instead of the dashboard running a range-vector aggregation every few seconds, the Prometheus server runs it once per interval and stores the output. Query latency drops, but the server pays CPU and storage write costs for every rule cycle.

Worked configuration

Suppose you expose http_requests_total, a counter (a cumulative metric that increases until reset) with labels job, handler, and instance. A dashboard currently runs:

sum by (job, handler) (rate(http_requests_total[5m]))

That expression touches every raw sample for every instance on each evaluation. A recording rule pre-aggregates by job and handler and drops instance:

groups:
  - name: http_aggregations
    interval: 30s
    rules:
      - record: job:http_requests_rate:rate5m
        expr: sum by (job, handler) (rate(http_requests_total[5m]))
        labels:
          job: "{{ $labels.job }}"
          handler: "{{ $labels.handler }}"

Save this in a rule file listed under rule_files in prometheus.yml, for example:

rule_files:
  - "/etc/prometheus/rules/http_aggregations.yml"

Apply the change by reloading Prometheus. On a Linux host running Prometheus under systemd, run:

sudo systemctl reload prometheus

Where to run: the Prometheus server host. Required permissions: read access to the rule file and permission to reload the Prometheus service or send SIGHUP. Placeholders: replace the path with your actual rule-file location. Expected check: after reload, the new metric appears in the expression browser. Risk: a syntax error in the rule file can prevent the group from loading; check /api/v1/rules for failure: true before relying on dashboards.

Verify the result

1. List rule groups and evaluation state:

curl -s http://PROMETHEUS_HOST:9090/api/v1/rules | jq '.data.groups[] | select(.name=="http_aggregations")'

Look for interval and confirm rules are not marked failure: true.

1. Compare the recording metric against an equivalent ad hoc query over the same lookback window. If they diverge, check scrape alignment and counter resets.
2. Alert on prometheus_rule_evaluation_failures_total and review prometheus_rule_evaluation_duration_seconds for slow groups.

Limits and common mistakes

1. Output cardinality

Keeping instance, pod, or request-path labels in labels: creates one output series per unique value. That can exceed the cardinality of the raw metric and cause storage pressure. Drop or aggregate high-cardinality dimensions before writing, as the example does by summing over instance implicitly through sum by (job, handler).

2. Counter resets

Recording rules do not smooth counter resets. Always wrap counters in rate() or increase() inside the rule expression. Recording a raw counter produces artificial spikes after service restarts.

3. Scrape and rule alignment

If the rule interval is shorter than the scrape interval, the rule can evaluate before the latest sample arrives, leaving dashboards with stale or incomplete aggregations. Set the rule interval at or above the scrape interval for the input metric.

4. Evaluation cost

Complex regex matching across millions of series inside one rule can block that rule group and delay other groups. Keep rule expressions scoped to the labels they need.

Rollback

Adding a recording rule changes server state because Prometheus begins writing new time series. To roll back, remove the rule file or comment out the rule, reload Prometheus, and confirm the group disappears from /api/v1/rules. Existing recorded samples remain until the retention period removes them; they are not deleted automatically by unloading the rule.

Practical check

Query the recording metric in the expression browser and compare its shape to the original ad hoc aggregation over the same window. If the lines match and dashboard load drops, the rule is working as intended.