How does Grafana's LRU query cache handle cache key fragmentation from high-cardinality Prometheus labels?

Short answer

Grafana's LRU query cache does not have a specialized mechanism to handle fragmentation from high-cardinality labels; it treats every unique query string as a distinct entry. When labels like request_id or session_id are interpolated into a query via template variables, they create unique cache keys that are rarely reused. Because the LRU (Least Recently Used) policy evicts the oldest entries once the per-data-source memory limit is reached, these one-time keys rapidly displace genuinely reusable data, leading to cache churn and a collapse in the hit rate.

Likely explanation vs. confirmed behavior

Confirmed behavior: The cache key is a composite of the exact query string, the time range, and the current values of template variables. Eviction is strictly LRU based on a memory cap per data source. High-cardinality interpolation ensures that the number of unique keys grows linearly with the number of unique label values, forcing frequent evictions.

Likely but version-dependent: While the core LRU logic is stable, the exact string generation for keys may vary between Grafana versions (e.g., changes in the Grafana 11 query editor). These internals are not fully documented, so behavior should be verified against your specific release via empirical testing.

How to tell fragmentation is degrading performance

Since Grafana lacks a native UI for cache hit rates, use these three signals to identify fragmentation:

1. Debug logging: Temporarily increase the log level for the caching component to observe hit/miss ratios. Note that debug logs may contain sensitive label values:

   [log]
   level = info
   
   [log.filters]
   caching = debug

2. Cache A/B testing: Execute a representative set of dashboard loads with the query cache enabled, then disable it in the configuration and repeat. If the p95 load times are nearly identical, the cache is likely churning and providing no benefit.
3. Process memory monitoring: Track the Resident Set Size (RSS) of the Grafana process. If memory climbs to the configured limit and plateaus while dashboard performance remains poor, the cache is saturated with fragmented entries.

Monitoring without exposing sensitive metrics

To avoid leaking sensitive label values through /api/admin/stats or logs, use these scoped approaches:

• Prometheus-side metrics: Monitor prometheus_tsdb_head_series and query duration histograms. High query volume from Grafana despite a "warm" cache indicates cache failure.
• Aggregated metrics: If your version exposes aggregated cache counters (hits/misses) via /metrics, scrape these into a separate monitoring instance. Aggregates do not contain the sensitive query strings.
• Synthetic Probes: Use a headless browser to measure end-to-end dashboard load latency. This provides a black-box signal of performance degradation without accessing internal cache keys.

Practical Resolution

Increasing the cache size only delays the problem if cardinality is unbounded. The resolution is to reduce cardinality before the query reaches Grafana: use Prometheus recording rules to aggregate data, remove high-cardinality labels from queries, or avoid using them in template variables. To prevent one noisy data source from starving others, isolate it into its own data source definition to leverage the per-source memory limit.

To provide a more specific recommendation, please clarify your Grafana version and whether the cardinality stems from template variable interpolation or the underlying Prometheus series.