Speed Up Prometheus Dashboards with Recording Rules

1. The Problem – Repeated Heavy Queries

Every time you open a Grafana panel or an alert fires, Prometheus evaluates the PromQL expression you wrote. If that expression is a long‑window rate() over many labels, the same expensive work is done over and over. Even a single dashboard can hit the server dozens of times per minute, turning a fast query into a bottleneck.

2. Recording Rules as a Materialized Cache

Recording rules let you pre‑compute a query once and store the result as a normal time series. Dashboards and alerts can then read the recorded series instead of re‑running the original expression. Think of it as a lightweight, on‑disk cache that Prometheus updates on a schedule.

3. How to Write a Recording Rule

A recording rule lives inside a recording rule group. The group has a name and an evaluation interval. Each rule has a record name and an expr.

groups:
- name: http
  interval: 30s
  rules:
  - record: job:http_requests:rate5m
    expr: sum by (job) (rate(http_requests_total[5m]))

• interval controls how often the rule is evaluated. Pick a value that balances freshness against load.
• The record name becomes the new metric name. Follow the level:metric:operation convention so you can search it easily.
• Every label that appears in the by clause becomes part of the stored series key. Keep the set of labels narrow to avoid exploding cardinality.

After adding the file to prometheus.yml or a separate rules/*.yml directory, reload Prometheus and verify:

promtool check rules /path/to/rules/
# Expect: OK

In the Prometheus UI, go to Status > Rules and confirm the group interval, last evaluation time, and health state.

4. Worked Example – HTTP Request Rate

Suppose you have a counter http_requests_total with labels job, status, method. A common metric is “requests per second per job” over a 5‑minute window. The raw query is:

sum by (job) (rate(http_requests_total[5m]))

Without a recording rule, every Grafana panel that shows this will re‑evaluate the rate() and the sum. With the rule above, the result is pre‑computed once every 30 s. Your panel can now simply query:

job:http_requests:rate5m

To verify the rule works, run both queries over the last hour and compare the series:

# Original
sum by (job) (rate(http_requests_total[5m]))

# Recorded
job:http_requests:rate5m

Both should return the same values for each job label. If they differ, check that the rule’s by clause matches the original.

5. Trade‑offs, Limitations, and Caveats

• Staleness: The recorded series lags by roughly scrape_interval + rule_interval. If you need sub‑second freshness, a recording rule may be too slow.
• Cardinality: Every unique combination of labels in the by clause becomes a stored series. A broad by (method, status) can create thousands of series per job.
• Evaluation Cost: Large rules consume CPU and memory on the Prometheus server. Monitor prometheus_rule_evaluation_duration_seconds after adding a rule.
• Rule Health: A failed evaluation leaves gaps. Use the UI to watch for Health: Error states.
• Non‑Deterministic Expressions: Avoid label_replace or random functions in recording rules; they can produce unpredictable series.

6. Checklist & Next Steps

1. Identify the slowest queries in your dashboards and the most frequently fired alerts.
2. Write a recording rule that pre‑computes the deterministic aggregate.
3. Validate locally with promtool check rules and in the UI.
4. Deploy, then monitor rule evaluation duration and series cardinality.
5. If load spikes, tighten the by clause or increase the rule interval.
6. Periodically audit recorded series to ensure they’re still needed; prune obsolete rules.

By treating recording rules as a lightweight, server‑side cache, you can keep dashboards snappy and alerts timely without rewriting every query. Start small, monitor, and iterate.