Precompute Expensive PromQL with Prometheus Recording Rules

The problem: dashboards that recompute the same expensive query

You open a dashboard that shows request latency percentiles across every service, namespace, and endpoint. The PromQL behind it aggregates millions of raw samples on every refresh. Prometheus CPU spikes, the query takes seconds, and the team starts asking for a faster database.

The fix isn't a faster database—it's moving that aggregation out of the query path. Prometheus recording rules evaluate a PromQL expression on a fixed schedule and write the result as a new time series. Dashboards and alerts then query the precomputed metric, which is a simple instant-vector lookup.

How recording rules work

A rule file defines groups of rules. Each rule has a name (the output metric), a PromQL expression, and an optional set of labels to add or overwrite. Prometheus evaluates the group at its interval (default matches the global evaluation_interval, which is 1m unless configured otherwise). The output series carry the rule name as their metric name and inherit labels from the source query, plus any labels you explicitly set.

Because evaluation happens on a fixed schedule, the cost is paid once per interval regardless of how many dashboards or alerts read the result. The trade-off is storage: every output series lives under the normal retention policy alongside scraped metrics.

When they help—and when they don't

• Good fit: Stable, frequently reused expressions—aggregations over many series, complex subqueries, or calculations that multiple dashboards and alerts share.
• Bad fit: Ad-hoc queries, expressions that change often, or anything over a high-cardinality label set (e.g., pod, request_id). The resulting series multiplication can pressure memory, disk, and compaction.
• Not a fix for: Poor label design, missing relabel_configs, or retention policies that are too long. Recording rules amplify the existing data model; they don't replace it.

Worked example: precomputing a service-level error rate

Suppose you have a counter http_requests_total with labels job, service, namespace, status. A dashboard shows the 5xx error rate per service across all namespaces, refreshed every 30s. The raw query:

sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m]))

That's two range-vector scans over high-cardinality data per dashboard load. Create a rule file recording-rules.yml:

groups:
- name: service-error-rate
  interval: 30s
  rules:
  - record: service:http_requests:rate5m
    expr: sum by (service) (rate(http_requests_total[5m]))
  - record: service:http_requests_5xx:rate5m
    expr: sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  - record: service:http_error_rate:ratio5m
    expr: service:http_requests_5xx:rate5m / service:http_requests:rate5m

Before Prometheus can evaluate these rules, add the file to the rule_files list in prometheus.yml:

rule_files:
  - 'recording-rules.yml'

The dashboard now queries service:http_error_rate:ratio5m—a single instant-vector lookup. The rule group runs every 30s, so the data is at most 30s stale.

Validating and monitoring the rules

1. Run promtool check rules recording-rules.yml before deploying. It catches syntax errors and missing metric names.
2. Reload Prometheus (SIGHUP or POST /-/reload). After one evaluation interval, query the new metric names to confirm they appear.
3. Inspect /api/v1/rules to see each group's lastEvaluation, evaluationDuration, and any errors. A group that consistently takes longer than its interval will cause stale data.
4. Compare dashboard query latency and Prometheus CPU before and after. Expect lower query latency; CPU may shift from query workers to the rule evaluator.

Trade-offs you'll live with

• Storage growth: Each output series consumes disk for the full retention period. In the example, three new series per service. If you have 500 services, that's 1,500 extra series—usually trivial, but it adds up with high-cardinality sources.
• Freshness delay: Data reflects the state at the last evaluation. If the interval is 30s, a spike that occurs right after evaluation won't appear until the next run.
• No backfill: Changing a rule expression only affects future evaluations. Historical values for the new metric name don't exist.
• Silent failures: If a rule group errors or exceeds its interval, dependent dashboards show stale data without a query-time error. Alert on prometheus_rule_evaluation_duration_seconds and prometheus_rule_evaluation_failures_total.

Next step: add one rule group this week

Pick the slowest dashboard query that's shared across at least two dashboards or alerts. Write a recording rule for its core aggregation, validate with promtool, deploy, and measure the dashboard load time after one evaluation interval. If latency drops and rule evaluation stays well under the interval, you've bought headroom for the next expensive query.