Using Prometheus Recording Rules to Reduce Dashboard Query Load

The problem: slow dashboards caused by heavy PromQL

When a dashboard repeatedly runs queries that contain expensive functions like rate() over long ranges or histogram_quantile(), each refresh forces Prometheus to recompute the same aggregation over and over. This can increase query latency, raise CPU usage on the Prometheus server, and make the user experience feel sluggish.

What recording rules do

A recording rule tells Prometheus to evaluate a PromQL expression at a fixed interval and store the result as a new time series. The generated series can then be queried directly, turning a costly computation into a simple instant‑vector lookup.

Recording rules are defined in YAML files that are loaded when Prometheus starts or can be added/removed via the HTTP API without a restart. The rule syntax looks like:

groups:
  - name: example.rules
    rules:
      - record: job:http_requests_rate:rate5m
        expr: rate(http_requests_total[5m])

The record field names the new series; the expr field holds the expression to pre‑compute. All original labels from the matched series are preserved unless you explicitly modify them with without or group_by.

Worked example: pre‑computing a five‑minute request rate

Suppose you have a dashboard that shows the request rate per job and you notice the query rate(http_requests_total[5m]) takes several seconds to return.

1. Create a rule file, e.g. /etc/prometheus/rules/request_rate.yml, with the content shown above.
2. Tell Prometheus to load the file. If you are using a static configuration, add the file path to the rule_files list in prometheus.yml and restart the server. If you prefer zero‑downtime updates, you can POST the rule to /api/v1/rules.
3. After the rule is active, query the new series: job:http_requests_rate:rate5m. This returns an instant vector that reflects the same values you would get from the on‑fly expression, but the computation has already been done.
4. In your dashboard panels, replace the original expression with the recorded series name. The panel now reads a single series instead of evaluating a range vector each time.

You can verify equivalence by comparing the two queries over a few time ranges; they should match within the expected scrape resolution.

Trade‑offs and limitations

While recording rules can reduce per‑query CPU, they introduce two costs:

• Storage overhead: each generated series adds samples to the TSDB. If you create many high‑cardinality rules, disk usage can grow noticeably.
• Evaluation overhead: the rule engine runs the expression at the configured interval (default is the global evaluation interval). Very short intervals or very complex expressions can add constant CPU load that may outweigh the query savings.

To keep the trade‑off favorable, limit the scope of rules with precise label matchers (e.g., {job=\"api\"}) and choose an interval that matches the granularity you need for dashboards (often 30s or 1m). Monitoring the metric prometheus_rule_evaluation_duration_seconds helps you spot rules that are taking too long to evaluate.

Checking that the rule works and rolling back if needed

After loading a rule, you can confirm it is active:

• In the Prometheus UI, go to Status → Rules and look for the rule name under the appropriate group.
• Query the metric prometheus_rule_evaluation_duration_seconds{rule=\"job:http_requests_rate:rate5m\"} to see recent evaluation times.

If you decide the rule is not providing enough benefit, you can roll back by removing the rule file (or deleting the rule via the API) and reloading Prometheus. Because the rule only adds series, removing it stops further samples from being written; the existing samples will remain until they age out according to your retention policy.

Actionable closing

Recording rules are a straightforward way to move repetitive, expensive PromQL work out of query time and into a scheduled evaluation step. Start with a single, high‑impact metric—like a request rate or latency histogram—measure the storage and evaluation impact, then expand only if the trade‑off remains positive. Keep an eye on the rule evaluation duration and TSDB growth, and you’ll have dashboards that feel more responsive without over‑provisioning your Prometheus instance.