What memory overhead does PromQL query evaluation impose on Prometheus instances?

Answer to the Core Question

Prometheus query evaluation can consume a significant amount of Go heap, especially for high‑cardinality metrics and long lookback windows. The peak memory footprint is largely driven by the number of matching series, the number of samples per series in the evaluated window, and the cardinality of the output series after aggregations or joins. In practice, a single complex query can spike the heap by several hundred megabytes, even if the steady‑state usage is modest.

Dominant PromQL Constructs and Their Heap Impact

• High‑cardinality joins (e.g., group_left / group_right, label_join, label_replace) materialize intermediate series sets in memory.
• Subqueries with large lookback windows (e.g., rate(metric[30d])) create a sample matrix that grows linearly with both series count and samples per series.
• Aggregations such as sum, avg, quantile_over_time build a map from label set to accumulator; the map size equals the output series count.
• Range vector selectors over long durations add 16‑byte data points per sample (float64 + int64) plus Go slice overhead.

Approximate Heap Formula

While Prometheus does not publish an exact formula, a practical estimate is:

peakHeap ≈ (matchingSeries × samplesPerSeries × 24 bytes) + (outputSeries × mapEntryOverhead) + queryASTOverhead

Typical values: mapEntryOverhead ≈ 48 bytes; queryASTOverhead ≈ 1–2 KB. A safety factor of 2–3× is common for bursty workloads.

Table: Rough Per‑Construct Overhead

Construct	Memory Pattern	Typical Peak Impact
High‑cardinality join	Intermediate series set	Series × samples × 24 bytes
Subquery (30d rate)	Sample matrix	Series × 30 days × 3 h × 24 bytes
Aggregation (by (le, instance))	Accumulator map	OutputSeries × 48 bytes
Range vector selector	Sample slice	Series × samples × 16 bytes

Query Cache: What It Does and Does Not Do

The query.cache.enabled setting stores only the final result matrix for a query. It reduces CPU and repeated allocation costs for repeated executions, but it does not bound the peak heap during the first evaluation that populates the cache. Cached queries can still trigger large memory spikes when first run.

Additionally, the cache has a per‑entry size limit (default 1 MB) and a total size limit; queries that produce larger results are not cached at all, so cache‑hit rates for heavy queries may be negligible.

Cached vs. Non‑Cached Execution Paths

• Non‑cached: The executor builds all intermediate data structures, performs allocations for each sample, and holds them until the result is written. Peak Go heap can spike to several hundred megabytes for large queries.
• Cached: On a cache hit, the executor simply copies the cached matrix into the response; heap usage drops to a few megabytes. However, the first hit still incurs the full allocation cost.

Practical Steps to Measure and Plan Capacity

1. Profile a representative query:

   go tool pprof -alloc_space http://localhost:9090/debug/pprof/heap
   

   This captures the allocation profile before and after the query.
2. Enable query logging:

   query.log.enable=true
   

   Correlate query_duration_seconds with process_resident_memory_bytes in the same scrape interval.
3. Test with and without cache:

   query.cache.enabled=false   # run query set
   query.cache.enabled=true    # run same query set again
   

   Measure peak RSS via cgroup memory.max or ps -o rss.
4. Generate synthetic data that mirrors production cardinality using promtool tsdb create-blocks and run the exact query set to validate the estimator.
Apply safety factor: Add 2–3× the measured peak to the target memory allocation in the Prometheus configuration (e.g., storage.max_local_storage_size or process.max_concurrent_queries).

Missing Diagnostic Detail

To refine the capacity plan, please provide the approximate cardinality of the metrics being queried and the typical lookback window for your most expensive queries. This data will tighten the estimator and help decide whether to enable the v3 query engine or adjust GC settings.