Prometheus Recording Rules: Precompute Expensive Queries Without Blowing Up Cardinality

The Problem: Dashboards That Time Out

You've built a dashboard that shows request latency percentiles across 500 Kubernetes pods. The PromQL query uses histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)). Every refresh spins up a query that scans millions of raw samples, and the UI hangs for 30 seconds. Your on‑call engineer gets paged because the alert evaluation itself exceeds the timeout.

Recording rules solve this by evaluating expensive expressions on a schedule and writing the results as new, pre‑aggregated time series. Dashboards and alerts then query the lightweight recorded metric instead of re‑crunching raw data every time. But each rule creates an active series that consumes CPU, memory, and storage—just like any scraped metric. Used carelessly, they shift the bottleneck from query time to ingestion and retention.

Design Rules That Reduce Cardinality

The cardinality of a recorded metric is determined by the label set emitted by its expr. If your expression groups by instance or pod, you haven't reduced fan‑out—you've just moved it. Effective rules aggregate high‑cardinality raw series (per‑pod counters) into lower‑cardinality summaries (per‑job or per‑cluster rates).

Use aggregation operators with by or without clauses to control the output label set explicitly:

groups:
- name: http-rollups
  interval: 1m
  rules:
  - record: job:http_requests:rate5m
    expr: sum(rate(http_requests_total[5m])) by (job)
  - record: job:http_request_duration_seconds_bucket:rate5m
    expr: sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)

The first rule collapses per‑instance counters into a single series per job. The second preserves the le label required for histogram quantile calculations while dropping instance. Both rules run every minute (interval: 1m), independent of the global scrape interval.

Run promtool check rules /etc/prometheus/rules/http-rollups.yml on your CI pipeline (no special permissions, just read access to the rule file) to catch syntax errors and references to metrics that don't exist in your cluster before deployment.

Evaluation Intervals and Chaining Latency

Each rule group has its own interval. Groups run in parallel; rules within a group run sequentially. A common pattern chains rules for multi‑stage rollups:

groups:
- name: instance-level
  interval: 30s
  rules:
  - record: instance:http_requests:rate1m
    expr: rate(http_requests_total[1m])
- name: job-level
  interval: 1m
  rules:
  - record: job:http_requests:rate5m
    expr: sum(instance:http_requests:rate1m[5m:1m]) by (job)

The job‑level rule reads the instance‑level recorded metric. Because the first group runs every 30s and the second every 60s, the job‑level metric reflects data at least 90 seconds old (30s evaluation + 60s interval). If you alert on job:http_requests:rate5m > 1000, the alert fires up to 90 seconds later than an alert on the raw expression. For latency‑sensitive alerts, keep the chain short or run the final rule at the same interval as its dependency.

Misaligned intervals cause silent gaps. If the instance group runs at 30s and the job group at 45s, their evaluation windows rarely align, and the job rule may see stale or missing samples. Stick to multiples (30s → 1m → 5m) and verify with prometheus_rule_group_iterations_missed_total—a rising counter means the group can't finish within its interval.

Operational Trade‑offs You Can't Ignore

Storage growth: Every recorded series is compacted and retained per your --storage.tsdb.retention.time. A rule that emits 10,000 series (e.g., by (job, pod, path)) adds the same write amplification as scraping 10,000 new targets. Before adding a rule, estimate cardinality: count by (__name__)({__name__=~\"job:http.*\"}) in the Prometheus UI.

Missing data ≠ zero: If rate(http_requests_total[5m]) returns no samples for a job (no traffic), the recorded metric simply has a gap. It does not write a zero. Downstream alerts like job:http_requests:rate5m == 0 won't fire. To record explicit zeros, wrap the expression: expr: sum(rate(http_requests_total[5m])) by (job) or vector(0).

Remote write gaps: Recording rule metrics are local to the Prometheus instance. They don't automatically appear in Thanos, Cortex, or Mimir unless your remote_write config includes them. Add a write_relabel_configs rule to keep the recorded metric prefix:

remote_write:
- url: https://long-term-storage/api/v1/write
  write_relabel_configs:
  - source_labels: [__name__]
    regex: 'job:.*'
    action: keep

Silent failures: Division by zero, missing labels, or malformed expressions log errors but don't stop other rules. Monitor prometheus_rule_evaluation_duration_seconds (alert if p99 > 50% of interval) and prometheus_rule_evaluation_failures_total to catch stale metrics before dashboards break.

Worked Example: Precomputing Histogram Quantiles

Your team needs p99 latency per service for an SLO dashboard. Raw histogram buckets have labels le, job, instance, path—high cardinality. You create two rules:

groups:
- name: histogram-buckets
  interval: 1m
  rules:
  - record: job:http_request_duration_seconds_bucket:rate5m
    expr: sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)
- name: slo-quantiles
  interval: 1m
  rules:
  - record: job:http_request_duration_seconds:p99
    expr: histogram_quantile(0.99, job:http_request_duration_seconds_bucket:rate5m)

The first rule aggregates buckets to per‑job, preserving le. The second computes the quantile on the pre‑aggregated buckets. The dashboard now queries job:http_request_duration_seconds:p99—a single series per job—instead of scanning raw buckets.

Verify the recorded series exist and have expected cardinality: count by (job) (job:http_request_duration_seconds_bucket:rate5m) should match your number of jobs, not instances × paths.

Actionable Checklist Before You Merge

1. Run promtool check rules in CI for every rule file change.
2. Estimate output cardinality with a test query against a representative Prometheus snapshot (promtool query instant works offline).
3. Set group interval to a multiple of the scrape interval; avoid prime numbers.
4. Add or vector(0) if downstream alerts depend on explicit zeros.
5. Confirm remote_write relabeling includes your recorded metric prefixes.
6. Deploy, then watch prometheus_rule_evaluation_duration_seconds and prometheus_rule_group_iterations_missed_total for 24 hours.

Recording rules are a lever, not a free lunch. They trade query‑time CPU for write‑path CPU and storage. Used with disciplined label design and interval alignment, they make high‑cardinality workloads queryable. Used as a band‑aid for every slow dashboard, they turn your Prometheus into a write bottleneck. Start with the one query that pages you at 3 AM, measure the impact, then expand.