Reduce Prometheus Query Load with Recording Rules

Dashboards stall when PromQL scans thousands of series

You have http_requests_total exported by every pod, instance, and endpoint. A cluster-wide traffic graph runs sum(rate(http_requests_total[5m])) by (job). Every refresh forces Prometheus to locate every matching series, calculate per-series rates, and aggregate them. With high cardinality—hundreds or thousands of label combinations—that work repeats on every dashboard load.

Recording rules move that computation from query time to ingestion time. Prometheus evaluates the expression on a schedule and stores the result as a new metric. Your dashboard then reads one precomputed series instead of thousands of raw ones.

What a recording rule does

A recording rule is a named PromQL expression evaluated by the Prometheus server at a fixed interval. The result is written as a new time series that you can query, graph, or alert on like any other metric. Rules live in YAML files loaded through the rule_files setting in prometheus.yml.

High cardinality means many unique label combinations. Aggregating thousands of series at query time is expensive; precomputing the aggregation shifts that cost to the write path.

Worked example: cluster-wide request rate

Create /etc/prometheus/recording_rules.yml:

groups:
  - name: cluster_aggregations
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

record names the new metric. expr is the PromQL to precompute. Every 30 seconds, Prometheus evaluates it and writes job:http_requests:rate5m.

Load the file in prometheus.yml:

rule_files:
  - /etc/prometheus/recording_rules.yml

Validate before reloading:

promtool check rules /etc/prometheus/recording_rules.yml

Run promtool on any machine that can read the file. It does not require a running Prometheus instance. Fix any syntax errors first.

Reload Prometheus. On a systemd host, use the local reload API if it is enabled:

curl -X POST http://localhost:9090/-/reload

This requires the --web.enable-lifecycle flag. If lifecycle is disabled, send SIGHUP to the Prometheus process instead. A malformed rule file can cause the reload to fail, so validate first.

Verify the result

Open the Prometheus Expression Browser and compare:

1. sum(rate(http_requests_total[5m])) by (job)
2. job:http_requests:rate5m

The second query should return far fewer series. Actual rendering speed depends on your data volume and hardware.

To confirm the rule produced data, query job:http_requests:rate5m. If nothing appears, check that raw series matching http_requests_total exist and that the rule file loaded without errors.

Prometheus exposes rule evaluation duration and failure metrics. Exact metric names vary by release, so confirm the metric for your version before alerting on it. A sustained increase in evaluation duration can signal that a rule is becoming expensive.

Trade-offs and limits

• Storage: every rule creates new time series. A rule with many output label combinations can grow disk usage significantly.
• Freshness: data is only as recent as the last evaluation. A 30-second rule can lag raw ingestion by up to 30 seconds.
• No backfill: rules do not compute values for time before they were added.
• Evaluation cost: rules with large lookback windows, predict_linear, or complex offsets can still be resource-intensive.

Rollback

Remove the rule file from rule_files in prometheus.yml and reload Prometheus. The server stops evaluating the rule. Previously recorded series remain in the TSDB until retention or compaction removes them; the delete API does not immediately reclaim disk space and requires the --web.enable-delete-api flag plus selector parameters.

Closing

Start with one expensive aggregation used across multiple dashboards. Record it, verify the series exists, and compare query behavior before adding more rules. Watch storage growth and rule evaluation duration so write-side savings do not create a new problem.