Using Prometheus Recording Rules to Speed Up Dashboards and Alerts

Why use recording rules

Recording rules let Prometheus evaluate a costly PromQL expression on a set schedule and store the result as a new time series. Dashboards and alerts that query the recorded series avoid recomputing the expression each time, reducing CPU load and query latency.

How a recording rule works

Define a rule group in a YAML file and reference it from prometheus.yml via the rule_files section. The group contains an optional interval (defaults to the global evaluation_interval) and a list of rules. Each rule specifies a record name and an expr to evaluate.

Example rule file

# file: /etc/prometheus/rules/latency.yml
groups:
  - name: latency_rules
    interval: 30s
    rules:
      - record: job:request_latency_seconds:mean5m
        expr: |
          avg by (job) (
            rate(http_request_duration_seconds_sum[5m]) /
            rate(http_request_duration_seconds_count[5m])
          )

Place this file on the Prometheus server, add it to prometheus.yml:

rule_files:
  - "/etc/prometheus/rules/*.yml"

After reloading Prometheus (kill -HUP or via the HTTP /-/reload endpoint), the rule evaluates the expression every 30 seconds and creates a new series named job:request_latency_seconds:mean5m with the job label from the aggregation.

Verification steps

1. Check that Prometheus loaded the file without errors: view prometheus_rule_loading_errors_total or the startup log.
2. Query the recorded metric: job:request_latency_seconds:mean5m in the expression browser; you should see samples spaced roughly by the rule interval.
3. Monitor rule performance: prometheus_rule_evaluation_duration_seconds histogram and prometheus_rule_evaluation_failures_total counter.

Limits and common mistakes

• Interval choice: Setting interval below 15 s increases write‑ahead log pressure and may cause the rule to fall behind scrape cycles. Keep it aligned with your scraping frequency.
• Cardinality explosion: Avoid recording raw per‑instance metrics without aggregation (e.g., omitting by or without clauses). Each unique label set becomes a separate series, raising memory usage and compaction load.
• Range vs interval mismatch: Using rate() or increase() with a range shorter than the rule interval creates gaps or double‑counted values. Ensure the range is at least as long as the interval (common practice: range ≥ 2 × interval).
• Rule dependencies: A recording rule can only reference another recording rule that appears earlier in the same file or in a previously loaded rule file. Cross‑file references require guaranteed load order; otherwise the dependent rule will fail to evaluate.
• Metric name changes: If you rename a recorded metric, old series are not rewritten; they remain until the retention period deletes them. Use metric relabeling to drop old series if needed.

Practical tip

Name recorded metrics with a trailing colon convention (e.g., job:request_latency_seconds:mean5m) to visually separate them from raw metrics and to avoid accidental clashes.