Cut Dashboard Latency with Prometheus Recording Rules: A Practical Guide

Why Your Dashboards Are Slow

When a Grafana panel takes seconds to render, the culprit is almost always a PromQL expression that forces Prometheus to scan millions of raw samples and perform heavy aggregations. Common patterns that trigger this include sum_over_time(), rate() over long windows, or multi‑label reductions on metrics that have dozens of unique label combinations. The result is a high CPU load on the Prometheus server and a poor user experience.

Recording Rules: Move the Work to Ingestion Time

A recording rule tells Prometheus to pre‑compute a PromQL expression at a regular interval and store the result as a new time series. Subsequent queries can then fetch that pre‑computed series instead of re‑calculating the expression each time.

Key points:

• Evaluation happens at the evaluation_interval defined in the global section of prometheus.yml or overridden per rule group.
• Each evaluation appends a new sample to the recorded series, so the rule must be written with the desired resolution in mind.
• Recording rules do not back‑fill existing data; they start collecting from the moment the rule is loaded.
• They can be defined in the same file as alerting rules; a reload will re‑evaluate all rules in the file.

Setting Up a Recording Rule

Below is a minimal example that pre‑computes the 5‑minute request rate for an API service. The rule will be evaluated every 30 seconds.

# /etc/prometheus/rules/api-recording.yml
groups:
- name: api-recording
  interval: 30s
  rules:
  - record: job:http_requests:rate5m
    expr: sum(rate(http_requests_total{job="api-server"}[5m]))

Steps to deploy:

1. Place api-recording.yml on a directory that the Prometheus process can read.
2. Reference the file in prometheus.yml:

   rule_files:
     - "rules/api-recording.yml"
   

3. Reload the configuration. If the /-/reload endpoint is enabled and authenticated, you can use:

   curl -X POST http://localhost:9090/-/reload
   

   Otherwise send a SIGHUP to the Prometheus PID.

Verifying the Rule Works

Prometheus exposes the status of all rules at /api/v1/rules or the Rules tab in the web UI. A healthy rule will show OK and a recent last_evaluation timestamp.

To compare performance, run the original expression and the recorded metric in the Expression Browser:

1. sum(rate(http_requests_total{job="api-server"}[5m]))
2. job:http_requests:rate5m

Both queries should return the same value, but the recorded metric will typically respond faster because it is a simple lookup. Keep in mind that the freshness of the recorded value depends on the evaluation interval; a 30s interval means the data can be up to 30 seconds out of date.

Trade‑offs and Practical Limits

While recording rules reduce query latency, they come with trade‑offs that you should monitor:

• Storage growth: Each rule creates an additional series. If the rule retains high‑cardinality labels (e.g., pod), the TSDB will store a separate series for every label combination, quickly inflating disk usage.
• CPU overhead: The rule evaluation itself consumes CPU proportional to the complexity of the expression and the number of series it touches. A very frequent evaluation interval can become a bottleneck.
• Data freshness: The recorded series lags behind raw data by up to one evaluation interval. For metrics that need millisecond precision, recording rules may not be suitable.
• Rule conflicts: If a recording rule and an alert rule share the same record name, the alert rule will take precedence and the recorded series will not be created.
• No retroactive data: Rules start collecting from the moment they are loaded; they do not compute values for historical data.

Mitigation strategies include:

• Drop unnecessary labels from the expression using label_replace or without clauses.
• Use a moderate evaluation interval (e.g., 30s-1m) that balances CPU usage and data freshness.
• Monitor prometheus_tsdb_storage_size_bytes and prometheus_tsdb_head_series metrics to detect runaway growth.
• Keep rule names unique and follow a convention like job:metric:operation to avoid accidental overrides.

Actionable Checklist

1. Identify the slowest PromQL queries by checking /api/v1/query response times or using Grafana’s query inspector.
2. For each candidate query, evaluate whether the result can be pre‑computed and whether the series will maintain acceptable cardinality.
3. Create a recording rule with an appropriate evaluation interval.
4. Reload Prometheus and verify the rule status.
5. Replace the original query in dashboards with the recorded metric and observe the latency improvement.
6. Continuously monitor TSDB size and CPU load; adjust the evaluation interval or drop labels if needed.

By following this workflow, you can systematically reduce dashboard latency while keeping an eye on storage and CPU costs.