Using Prometheus Recording Rules to Reduce Query Load and Speed Up Dashboards

Desired Outcome

When Prometheus receives many queries for the same expression, dashboards become sluggish and alerting rules may fire late. By defining a recording rule you let Prometheus pre‑evaluate the expression and store the result as a new time series. The new series can be queried directly, reducing CPU and memory usage on the server and making dashboards responsive, which in turn improves user experience and enables timely alerts. This approach is especially useful for high‑traffic environments where the same metric is queried repeatedly across many panels.

Prerequisites

• Prometheus version 2.0 or newer; rule reloading is stable from 2.0 onward.
• A prometheus.yml configuration that includes a rule_files section.
• File system permissions that allow the Prometheus process (often the prometheus user) to read the rule file.
• Network access to the Prometheus HTTP endpoint (e.g., localhost:9090) for verification commands.

Step‑by‑Step Procedure

1. Create the rule file

Place a new YAML file, for example recording_rules.yml, in a directory that the Prometheus user can read, such as /etc/prometheus/rules/. The file must contain one or more rule groups, each with a name: and a list of rules. Each rule specifies a record: name for the new metric and an expr: that defines the PromQL expression to be pre‑evaluated. The interval: field inside the group controls how often the expression is recomputed; the default is 1 minute, but it can be adjusted to match the dynamics of the metric.

# /etc/prometheus/rules/recording_rules.yml
 groups:
   - name: myapp
     interval: 1m
     rules:
       - record: up_myapp
         expr: up{job: myapp}

In this example the rule creates a series called up_myapp that represents the up metric filtered to the myapp job. Because the expression is pre‑evaluated, the Prometheus server no longer has to recompute the full expression for every query, which lowers query latency.

2. Reference the rule file in prometheus.yml

Add the path to the file under the global rule_files list. Example configuration:

global:
  scrape_interval: 30s

rule_files:
  - /etc/prometheus/rules/recording_rules.yml

After editing, the Prometheus process must reload the configuration so that it reads the new rule file. A reload does not restart the server; it only re‑parses the rule files and applies any changes.

3. Reload the rule file

The safest way to apply changes without stopping the server is to send a reload request to the HTTP endpoint:

curl -X POST http://localhost:9090/-/reload

If the Prometheus instance is protected by basic auth, include the credentials, e.g., curl -u admin:password -X POST http://localhost:9090/-/reload. Successful reloads are logged as rule_files reload and the API returns 200 OK. The reload is instantaneous for the rule files; the underlying scrape jobs continue unchanged.

4. Verify that the rule is active

Query the recorded metric directly via the HTTP API:

curl 'http://localhost:9090/api/v1/query?query=up_myapp'

The response should contain a series with the same label set as up{job: myapp} but only for that job. You can also open the Prometheus UI, go to Status → Rules, and confirm that the rule appears under the myapp group with a green check mark. This visual confirmation helps ensure that the rule was loaded correctly.

5. Adjust the evaluation interval if needed

The interval: value inside the rule group determines how frequently the expression is evaluated. Shortening it (for example to 30s) reduces staleness but raises CPU and memory consumption. To find a suitable balance, monitor the process_cpu_seconds_total and prometheus_tsdb_head_series metrics while testing different intervals. A common practice is to keep the interval at 1 minute unless the metric is highly dynamic, such as a rapidly changing counter.

Fine‑tuning tip

If you notice that the CPU usage spikes after a reload, try increasing the interval: to 2 minutes or more. Conversely, if dashboards feel too stale, you can lower the interval, but be aware that each additional evaluation adds load. The evaluation_interval metric exposed by Prometheus can be queried to see the actual interval used for rule evaluation.

Verification Checks and Relevant Recovery Options

• Syntax validation: Run promtool check rules /etc/prometheus/rules/recording_rules.yml before reloading. Any error will be reported in the Prometheus logs and the reload will be rejected, preventing a broken rule set from being applied.
• Rule loading verification: After reload, curl http://localhost:9090/api/v1/rules should list the up_myapp rule without syntax errors. The response includes the rule name, interval, and a status flag.
• Metric correctness: Compare a direct query up{job: myapp} with a query on the recorded series up_myapp. The values should be identical, confirming that the pre‑evaluation produced the expected result.
• Storage impact assessment: Use curl http://localhost:9090/api/v1/series?match[]=__name__=up_myapp and examine the number of series returned. A sudden increase may indicate high‑cardinality labels that need to be trimmed, as each unique label combination creates a new series.

Recovery & Rollback

If a syntax error prevents the reload from succeeding, Prometheus retains the previous rule set, so the system remains functional. To revert:

1. Rename or delete the problematic rule file.
2. Reload the configuration again with curl -X POST http://localhost:9090/-/reload.
3. Confirm that the old rules are still present by checking /api/v1/rules.

In the unlikely event that a full process restart is required (for example, after a concurrent change to scrape_interval), stop the Prometheus service, start it again, and verify that the rule file is loaded during startup. The restart itself acts as a rollback because the previous rule set is discarded.

Limitations and Practical Verification

Recording rules inherit all labels from the source expression. High‑cardinality label values, such as user IDs or request hashes, can cause the series count to explode, leading to storage pressure and slower queries. To avoid this, keep the label set small and consider aggregating on low‑cardinality dimensions before recording. For example, if you record http_requests_total{job="myapp", instance="host1", handler="login", user="alice"}, the cardinality can grow quickly; instead record http_requests_total{job="myapp", instance="host1"} and let higher‑level dashboards aggregate by handler or user as needed.

Very frequent evaluation intervals increase CPU and memory usage. If you observe sustained growth in process_cpu_seconds_total or memory consumption, increase the interval: value or remove the rule. The trade‑off is between query latency and resource consumption.

When using remote write or Thanos for long‑term storage, recorded series must be queried from the same Prometheus instance that generated them; otherwise the data may be missing from the queried time range. This limitation is important for multi‑cluster setups where each cluster runs its own Prometheus.

Practical verification: create a dashboard panel that queries up_myapp for the past hour. The panel load time should drop noticeably compared to a panel that queries up{job: myapp} directly. Additionally, check the evaluation_interval metric in the Prometheus UI to ensure the rule is being evaluated at the expected frequency, and monitor the rule_files reload logs for any errors after each change.