Optimizing Prometheus Dashboards with Recording Rules: A Practical Guide

Problem: Dashboards that Stall on Heavy Queries

When a Grafana panel or an ad‑hoc Prometheus query asks for a complex aggregation—such as a 30‑minute rate over thousands of pods—Prometheus must scan and compute the result each time. If the query runs every few seconds, the UI can feel sluggish or even time out during traffic spikes.

Thesis: Precompute with Recording Rules

Recording rules let you write a PromQL expression once, let Prometheus evaluate it on a schedule, and store the result as a new time series. Subsequent queries then read the pre‑computed value instead of re‑executing the heavy expression.

1. What Are Recording Rules?

In Prometheus, a recording rule is a YAML entry inside a rule_group. Each group has an interval that defines how often the rule runs. The expr field contains a standard PromQL expression that can use instant or range vectors. When the rule fires, Prometheus writes the result as a new metric, optionally adding a record name that follows the label_name:metric_name convention.

Example rule group snippet:

groups:
- name: http_errors
  interval: 1m
  rules:
  - record: job:http_error_rate:ratio_5m
    expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Key points:

• Rule files are loaded automatically when Prometheus starts or when a /-/reload request is sent.
• Rule evaluation happens in the background; it does not block query handling.
• Resulting metrics inherit the labels of the expression unless you explicitly drop them with without or on clauses.

2. Worked Example: 5‑Minute HTTP Error Rate

Assume your service exposes http_requests_total with a status label. The raw query is expensive if you run it every 15 seconds:

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Define a recording rule that precomputes the ratio:

groups:
- name: http_errors
  interval: 1m
  rules:
  - record: job:http_error_rate:ratio_5m
    expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

After Prometheus reloads the rule file, you can query job:http_error_rate:ratio_5m every few seconds with negligible latency.

3. Validation, Reload, and Verification

1. Validate the YAML before changing the running server:

   promtool check rules /path/to/rules.yml
   

   Running promtool is safe; it exits with a non‑zero status if the file is malformed.

2. Reload Prometheus without downtime:
   • Send SIGHUP to the Prometheus process, or
   • POST to http://YOUR_PROMETHEUS_URL/-/reload (requires web.enable-lifecycle set to true).
3. Check reload status via the status endpoint:

   curl -s http://YOUR_PROMETHEUS_URL/api/v1/status/tsdb
   

   Look for rules_freshness_seconds and any error messages.

4. Verify equivalence on a sample time range:
   • Run the original query and the recording rule metric in the Prometheus UI.
   • Compare the series; they should match for the same timestamp.

4. Trade‑offs and Caveats

• Cardinality Inflation: If the rule preserves high‑cardinality labels (e.g., pod_name), the new metric can duplicate a large amount of data. Use without or aggregate early to keep series count low.
• Storage Cost: Each recording rule writes a new series to the TSDB. Monitor series count with prometheus_tsdb_head_series.
• Reload Semantics: Reloading rules is not atomic. A syntax error in a new rule file will leave the old rules in place until the next successful reload, but it can also leave the new rule absent, causing a temporary data gap.
• Version Differences: Rule syntax and validation rules have evolved since Prometheus 2.30. Test your rule file against the exact version you plan to run.
• Evaluation Lag: If the rule interval is longer than the query refresh interval, dashboards will display step‑like data. Align intervals carefully.

5. Decision Matrix

Scenario	Raw Query?	Recording Rule?
Ad‑hoc debugging of a single pod	Yes	No
Executive dashboard with 30‑day trend	No	Yes
Alerting every 15 s on a 5‑min window	No	Yes (for stability)
Simple counter, low cardinality	Yes	No

Actionable Take‑away

Identify the most expensive queries that hit dashboards or alerts, write a recording rule for each, and set an evaluation interval that matches your refresh cadence. Validate with promtool, reload safely, and verify that the new metric matches the original expression. Keep an eye on series count and label cardinality to avoid TSDB bloat. With these steps, Prometheus can deliver fast, reliable metrics while keeping query latency under control.