Solving Dashboard Lag with Prometheus Recording Rules

The Problem: High-Cardinality Query Latency

In large-scale environments, dashboards often slow down or time out when visualizing aggregate trends. This typically happens when a PromQL query must process high-cardinality data—metrics with a vast number of unique label combinations—at request time. For example, calculating the total request rate across a Kubernetes cluster requires Prometheus to scan every individual pod series, compute the rate, and then sum them up.

If you have 1,000 pods each exporting a metric, a dashboard refreshing every 15 seconds forces Prometheus to perform these millions of calculations repeatedly. This consumes significant CPU and memory, leading to the "loading spinner" experience for users and potential timeouts for critical alerts.

Pre-computing with Recording Rules

Recording rules allow you to shift the computational burden from the query time (when a user opens a dashboard) to the evaluation time (a background process). A recording rule evaluates a PromQL expression at a regular interval and persists the result as a new, separate time series in the database.

By creating a "materialized view" of your data, you replace a complex aggregation with a simple lookup. Instead of asking Prometheus to sum 1,000 series every time the page refreshes, the dashboard simply reads the one pre-computed series that already contains the sum.

Implementation Example: Service-Level Aggregation

Consider a scenario where you need to monitor the 5-minute request rate per service, but the raw metric http_requests_total includes high-cardinality labels like pod and instance.

Create a rule file (e.g., /etc/prometheus/rules/aggregates.yml) with the following configuration. This assumes Prometheus v2.0 or later:

groups:
  - name: service_metrics
    interval: 60s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job, namespace) (rate(http_requests_total[5m]))

Applying the Configuration

To load these rules without restarting the server, you must have started Prometheus with the --web.enable-lifecycle flag. Run the following command from a terminal with network access to the Prometheus API:

# Send a POST request to trigger a configuration reload
curl -X POST http://<prometheus-host>:9090/-/reload

Risk: Reloading configuration via the API is generally safe, but ensure your YAML syntax is validated first using promtool check rules to avoid loading errors that could stop rule evaluation.

Trade-offs and Limitations

Recording rules are a performance optimization, not a free resource. There are two primary costs to consider:

• Storage Overhead: Every recording rule creates new time series. While these are usually lower cardinality than the raw data, excessive use of recording rules can lead to "metric explosion," increasing the disk space required for the Time Series Database (TSDB).
• Resolution Loss: The recorded metric is only as granular as the interval defined in the rule group. If your interval is 60s, any transient spikes occurring between those evaluations are smoothed over. For high-precision alerting, raw queries are still preferred; for long-term trends and dashboards, recording rules are ideal.

Verifying the Result

To confirm the rule is functioning and providing the expected performance gain, perform these checks in the Prometheus UI:

1. Rule Health: Navigate to Status → Rules. Verify that the service_metrics group is listed and that the last evaluation timestamp is current.
2. Data Validation: Run the raw expression sum by (job, namespace) (rate(http_requests_total[5m])) and the new metric job:http_requests:rate5m side-by-side in the Graph tab. The values should be nearly identical.
3. Latency Comparison: Observe the execution time in the browser's network tab or the Prometheus performance logs. The recorded metric query should return significantly faster than the raw aggregation.