Prometheus Recording Rules: Precompute Expensive PromQL Before Your Users Feel It

The Dashboard That Ruins Your Morning

You open a Grafana panel to check API error rates, and the spinner hangs for eight seconds. The query is not complicated—a ratio of 5xx requests to total requests—but it scans millions of raw samples across dozens of pods. Every refresh recomputes the same aggregation from scratch. Dashboards feel sluggish, alert rules evaluate slowly, and your Prometheus server burns CPU while users wait.

The useful takeaway: Prometheus recording rules let you schedule expensive PromQL expressions ahead of time and store the results as ordinary time series. Instead of computing an error ratio at query time, you precompute it on a fixed schedule and query a single, cheap metric. You move the cost from every dashboard load to a predictable background evaluation.

How Recording Rules Work

A recording rule is a PromQL expression that Prometheus evaluates on a schedule and writes back into its time-series database. You define rules inside a rule group. Each group has an interval—how often Prometheus runs the rules in that group—and each rule has a record name and an expr field containing the PromQL query.

Prometheus evaluates groups sequentially. If one expression in a group is extremely expensive, it can delay the rules that come after it in the same group. For this reason, it is usually better to place heavy rules in their own group with an interval that comfortably exceeds their evaluation time.

You load rules either by passing --rule-files=rules.yml on the Prometheus command line or by adding a rule_files block to prometheus.yml. After changing the file, reload Prometheus with a POST to http://<prometheus-host>:9090/-/reload or by sending SIGHUP to the process. You need filesystem access to the Prometheus host and permission to restart or signal the service.

A Worked Example: Error Ratios

Suppose your raw metric is http_requests_total with labels for job and HTTP status code. A typical error-ratio panel runs this expression:

sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

Every panel refresh and every alert evaluation repeats the same rate() calculations over the full five-minute window. If the api job runs on many pods, that is a lot of repeated work.

Instead, create a rule file (for example, /etc/prometheus/rules/api.yml) with the following content:

groups:
  - name: api-error-ratios
    interval: 30s
    rules:
      - record: job:http_errors:ratio5m
        expr: |
          sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))

After reloading Prometheus, the server evaluates that expression every 30 seconds and stores the result as a new series named job:http_errors:ratio5m. Your dashboard panel and your alert rules can now query a single series:

job:http_errors:ratio5m

The naming convention here is level:metric:operations. The job prefix tells you the series is aggregated per job, http_errors describes the measurement, and ratio5m records that it is a ratio computed over a five-minute window. A consistent convention helps operators distinguish raw metrics from precomputed ones at a glance.

Trade-offs You Should Measure

Recording rules are not free. They add storage and write load because every evaluation appends new samples. They also introduce a delay: if your group interval is 30 seconds, the recorded series can lag the raw data by up to 30 seconds. For alerting, that lag can reduce sensitivity to short spikes.

Another limitation is that recording rules do not backfill. If you create a new rule today, the recorded series starts empty. Historical dashboards will show gaps until enough time has passed. Conversely, if the source metric http_requests_total stops being scraped, the recorded series will eventually go stale. That staleness can mask an upstream ingestion problem if you only look at the precomputed metric.

Finally, avoid precomputing expressions that are rarely used. Each recorded series consumes disk space and increases write amplification on the Prometheus server. The practical approach is to profile your slowest queries, precompute the top offenders, and compare query latency and Prometheus load before and after.

Checking Rule Health

Treat the rule pipeline as a production workload, not a hidden cron job. Prometheus exposes internal metrics that tell you whether rules are succeeding and how long they take.

You can inspect loaded rules through the HTTP API. Run this from any machine that can reach the Prometheus server:

curl -s http://<prometheus-host>:9090/api/v1/rules | jq '.data.groups[] | select(.name=="api-error-ratios")'

Look for the health field of each rule and the evaluationTime of the group. A healthy rule reports health: "ok". If the group duration approaches the interval, evaluations may start to queue.

Prometheus also exposes self-monitoring metrics. Query these in the expression browser or via /api/v1/query:

• prometheus_rule_group_last_duration_seconds — how long the last evaluation of each group took.
• prometheus_rule_evaluation_failures_total — cumulative count of failed evaluations.

If prometheus_rule_evaluation_failures_total increases, the rule expression is likely invalid or the source metrics are missing. If prometheus_rule_group_last_duration_seconds trends upward, the expression is getting more expensive or the group interval is too tight.

Start Small and Verify

Pick one slow dashboard panel, convert its query into a recording rule, and measure the difference. Compare the raw expression and the recorded metric side by side in the expression browser to confirm they return equivalent values. Check the rule health metrics after a few evaluation cycles. If the measured benefit justifies the extra storage, expand to other expensive queries. If it does not, remove the rule and keep the raw query.