Stop Dashboard Timeouts: Optimizing PromQL with Recording Rules

The Cost of Complex Queries

You have a Grafana dashboard that looks great, but as your time range expands from 1 hour to 30 days, the panels start to spin or return "Query Timeout" errors. This happens because Prometheus is a pull-based system that calculates aggregations on the fly. When you ask for the 99th percentile of request latency across 500 pods over a month, Prometheus must scan millions of raw samples at query time.

The solution isn't always to throw more CPU at the Prometheus server. Instead, you can use Recording Rules to pre-compute these expensive calculations in the background and store the results as new, lightweight time series.

How Recording Rules Shift the Load

A recording rule takes a complex PromQL expression, evaluates it at a regular interval (e.g., every 15 seconds), and writes the result back into the Time Series Database (TSDB) as a new metric. This shifts the computational burden from the reader (the person loading the dashboard) to the writer (the Prometheus server's background process).

This is particularly effective for high-cardinality data—metrics with many unique label combinations. By aggregating these into a recording rule, you collapse thousands of series into a handful of pre-summed values, making long-term trend analysis nearly instantaneous.

Implementing a Recording Rule

Recording rules are defined in a YAML file and loaded into Prometheus via the --rule.files flag. To keep your metrics organized, it is standard practice to use a naming convention: level:metric:operation.

Consider a scenario where you frequently calculate the average CPU usage per cluster across multiple namespaces. Instead of running this query every time you refresh your dashboard:

avg(rate(container_cpu_usage_seconds_total[5m])) by (cluster)

You can define a recording rule in a file (e.g., rules.yml):

groups:
  - name: cluster_performance_rules
    interval: 1m
    rules:
      - record: cluster:cpu_usage:avg_5m
        expr: avg(rate(container_cpu_usage_seconds_total[5m])) by (cluster)

Deployment and Execution

• Permissions: Ensure the Prometheus process has read access to the rules.yml file.
• Configuration: Add the file to your startup command: --rule.files="/etc/prometheus/rules.yml".
• Verification: Run the Prometheus server and navigate to the Status > Rules page in the web UI to ensure the rule state is "OK".

Now, instead of the complex avg(rate(...)) query, your dashboard simply queries cluster:cpu_usage:avg_5m. The result is a single time series per cluster, reducing the data scanned from millions of points to a few hundred.

Trade-offs and Storage Costs

Recording rules are not a "free" performance boost. There are two primary trade-offs to consider:

• Storage Overhead: Every recording rule creates a new time series. If you create rules for every possible combination of labels, you may significantly increase your disk usage and memory pressure.
• Evaluation Lag: Because the rule runs on a defined interval, the pre-computed metric is only as fresh as the last evaluation. If your interval is 1 minute, your dashboard data may lag behind the raw metrics by up to 60 seconds.

Testing the Result

To verify the efficiency gain, use the Prometheus Expression Browser:

1. Run the original complex query and note the execution time (visible in the browser's developer tools network tab or via the Prometheus API).
2. Run the query for the new recording rule metric.
3. Compare the two. In high-cardinality environments, you will typically see a reduction in query latency from several seconds to a few milliseconds.

If the recording rule metric returns no data, check the Rules page for syntax errors or verify that the underlying raw metrics (e.g., container_cpu_usage_seconds_total) are actually being ingested.