When to Use Prometheus Recording Rules (and When Not To)

Your dashboard suddenly feels sluggish during peak hours. The PromQL query behind it scans months of raw metrics, and Prometheus has to aggregate on the fly. For teams running high‑cardinality environments, this isn’t just a nuisance—it’s a reliability risk.

Recording rules are a Prometheus feature that evaluates a PromQL expression at a configured interval and persists the result as a new time series in the TSDB. This means downstream queries and alerts hit a single precomputed series instead of recomputing the aggregation every time the dashboard refreshes.

A concrete example

Suppose you want to show a 5‑minute rolling average of CPU usage per container on a dashboard that updates every minute. Without a recording rule, Prometheus must scan all raw container_cpu_usage_seconds_total samples for the last 5 minutes each time the query runs.

groups:
  - name: recording-rules
    interval: 5m
    rules:
    - record: container_cpu_rate5m
      expr: rate(container_cpu_usage_seconds_total[5m]) by (container, instance)

In this rule, record: gives the new series a name container_cpu_rate5m. The expr: is the PromQL rate() function computing per‑container CPU seconds per second over a 5‑minute window. The group‑level interval: 5m tells Prometheus to reevaluate the expression every 5 minutes, matching the window size and reducing unnecessary work.

Place the rule file somewhere accessible to Prometheus, for example /etc/prometheus/rules.yml. Then add it to your prometheus.yml:

rule_files:
  - /etc/prometheus/rules.yml

After saving the file, reload the configuration without restarting Prometheus using the /-/reload endpoint:

curl -X POST http://localhost:9090/-/reload

Trade-offs and limitations

• Storage Overhead: Each recorded series consumes TSDB storage and head block memory just like a scraped metric. Unused rules increase retention costs.
• Cardinality Growth: Recording rules do not reduce raw ingestion cardinality; they add derived series on top of existing high‑cardinality data, which can explode your series count.
• Evaluation Gaps: A failing rule evaluation produces no series for that cycle rather than halting Prometheus, which can create gaps in dashboards.
• Configuration Risks: Syntax errors in a rule file prevent that rule group from loading. The reload endpoint reports the failure without rolling back other healthy groups.

Practical verification

To ensure your rule is working correctly and efficiently, follow these steps:

1. Check Configuration: Open /api/v1/status/config in the Prometheus UI to confirm rule files are loaded and paths are correct.
2. Verify Data: Query the exact recorded metric name (container_cpu_rate5m) in the expression browser after at least one evaluation interval to ensure data is appearing.
3. Monitor Health: Query prometheus_rule_group_last_duration_seconds and prometheus_rule_group_rules_evaluated_total to track rule performance and frequency.
4. Measure Impact: Compare the TSDB series count before and after enabling a rule to measure the actual storage impact.

Actionable takeaway: Start by identifying your most expensive dashboard query. Extract it into a recording rule with a matching interval, and monitor the series count to ensure you aren’t trading query speed for unsustainable storage costs.