Prometheus Recording Rules: Precompute Expensive Queries and Cut Dashboard Latency

The short answer

If a dashboard panel or alert runs a heavy PromQL expression like rate(node_cpu_seconds_total[5m]) across thousands of series, every refresh re-evaluates it from raw samples. A recording rule moves that work into the Prometheus server itself: it evaluates the expression on a fixed interval and stores the result as a brand-new time series. Your dashboard then queries a plain metric name and returns almost instantly.

Recording rules are the right tool when the same expensive expression appears in multiple dashboards or alerts, when range queries over long windows are slow, or when you want a stable, pre-aggregated metric for federation or long-term storage. They are not the right tool for one-off ad-hoc queries.

How they work

Prometheus loads rule files listed under rule_files in prometheus.yml. Each file contains groups; each group has an evaluation interval and a list of rules. On every tick of the interval, the server evaluates the rule's expr and writes the result into its local storage under the metric name given in record. From that point on, the recorded series behaves like any other metric — you can graph it, alert on it, or federate it.

Because evaluation happens server-side at a fixed cadence, the cost is paid once per interval regardless of how many dashboards or users query the result.

A working configuration

First, reference a rules file in prometheus.yml (edit as the user that owns the Prometheus config, typically requiring write access to the config directory and permission to reload the service):

rule_files:
  - /etc/prometheus/rules/node.yml

Then define the rule group in /etc/prometheus/rules/node.yml:

groups:
  - name: server
    interval: 30s
    rules:
      - record: job:node_cpu_seconds_total:rate5m
        expr: sum by (job) (rate(node_cpu_seconds_total[5m]))
      - record: job:node_memory_available_ratio
        expr: |
          avg by (job) (
            node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
          )

Apply the change by reloading Prometheus — either send SIGHUP to the process (kill -HUP <pid>) or POST to /-/reload if you started it with --web.enable-lifecycle. A full restart also works but interrupts scraping. Before reloading, validate the file with promtool check rules /etc/prometheus/rules/node.yml; it catches syntax errors and malformed expressions without touching the running server.

After one evaluation interval, query job:node_cpu_seconds_total:rate5m in the expression browser. It should return data immediately. Also open Status → Rules in the UI to confirm the group shows a recent "last evaluation" timestamp and no errors — this is the fastest way to catch silent failures.

Naming convention

The colon-separated pattern level:metric:operations (for example job:node_cpu_seconds_total:rate5m) is a widely used convention, not a requirement. It encodes the aggregation level and the operations applied, so anyone reading the name knows the series is a 5-minute rate aggregated to job level. Following it keeps rule files readable as they grow.

Limits and common mistakes

• Evaluation interval. Rules cannot meaningfully evaluate faster than your scrape interval — evaluating a 5-minute rate every 15s against a 60s scrape interval just recomputes near-identical results and adds load. A 30s or 60s group interval is a sane default for most setups. Very short intervals across many groups increase CPU and memory pressure and, in extreme cases, contribute to out-of-memory kills.
• Silent failures. A typo in the file path under rule_files, or a file Prometheus cannot parse, means the recorded metric simply never appears — dashboards show "no data" rather than an error. Always run promtool check rules and check Status → Rules after changes.
• Missing or duplicated record names. Every rule needs a record name, and two rules writing to the same name with overlapping label sets produce confusing results. Keep names unique per aggregation.
• Stale semantics. A recorded metric is only as fresh as the last evaluation. Alerting on a recorded series evaluated every 60s adds up to a minute of latency compared with alerting on raw metrics. For fast-firing alerts, evaluate the expression directly in the alerting rule instead.
• Cardinality. Recording rules that keep high-cardinality labels (instance, pod, container) multiply series count just like raw metrics. Aggregate away labels you do not need in the expr.

Verifying the result

Three quick checks confirm a rule is healthy: promtool check rules passes before reload; Status → Rules shows the group with a fresh last-evaluation time and zero errors; and a direct query of the recorded name returns series with the expected labels. If the query returns nothing but the rule looks healthy, confirm the underlying expression returns data on its own — a rule recording an empty result produces no series at all.