Grafana Dashboard Variables with Prometheus: A Practical Guide to Multi-Service Dashboards Without Query Explosions

The Dropdown That Worked Yesterday

You add a service dropdown to a Grafana dashboard so a single view can serve multiple teams. It works fine with five services. Then someone deploys a new batch of microservices, the dropdown returns two thousand values, the dashboard takes half a minute to render, and Prometheus starts timing out. The feature did not change; the cardinality did.

Thesis: Grafana dashboard variables populated from Prometheus labels are a supported, long-standing pattern for multi-service dashboards, but their cost is invisible until high-cardinality labels or panel repetition multiply queries. Treat variable design as a query-budget decision, not a UI convenience.

What a Prometheus Variable Actually Costs

A dashboard variable of type Query can run a data-source query to populate its options. With a Prometheus data source, the common helper is label_values(<metric>, <label>), which returns every value that label currently holds for the metric. Grafana then substitutes the selected value into panel queries, usually as a regex match such as service=~"$service".

The danger is unbounded cardinality. Labels like instance, pod, path, or user_id can have thousands or millions of distinct values. A variable built on one of those labels forces Grafana to fetch and render a massive option list, bloats the dashboard URL, and can impose real load on Prometheus. Exact query helpers and UI labels vary by Grafana version, so validate syntax in your own instance before saving.

Worked Example: A Curated Variable Chain

Instead of one flat dropdown, build a hierarchy: cluster → namespace → service. Each variable filters the next, keeping option lists short and meaningful.

Before creating the variable, check the cardinality of the label you plan to expose. Run this in a terminal with network access to your Prometheus server; replace <prometheus-url> with your actual endpoint:

curl -sG "<prometheus-url>/api/v1/query" \
  --data-urlencode 'query=count(count by(service)(up))'

Expected check: The JSON response contains a single numeric value. If that number is in the thousands, do not use service as a flat dropdown. If it is manageable—say, under a few hundred—proceed.

In Grafana, open Dashboard settings → Variables → New variable and configure:

• Name: service
• Type: Query
• Data source: <your-prometheus-datasource>
• Query: label_values(up{cluster="$cluster", namespace="$namespace"}, service)
• Multi-value: Off
• Include All: Off

Then reference it in a panel query:

sum by (service) (rate(http_requests_total{cluster="$cluster", namespace="$namespace", service=~"$service"}[$__rate_interval]))

If you want one panel per service, set Repeat by variable to service on the panel. Grafana will render a copy of the panel for each selected value.

The Fan-Out Problem

Multi-value variables and Include All are convenient, but they multiply work. When a panel repeats by a variable, Grafana issues one query per repetition. Ten repeated panels across fifty services is five hundred simultaneous queries, not ten. Multi-value without repetition is cheaper—the regex expands to service=~"svc1|svc2|..." in a single query—but the query itself becomes larger and slower as the list grows.

Practical check: Open the dashboard, select one value, then several, then All. Use Inspect → Panel stats → Query on a repeated panel and record the query count and total request time. If the count scales linearly with your selection, constrain the time range or narrow the variable hierarchy before enabling All.

Variables Are Exploration Filters, Not Alert Boundaries

A common mistake is treating a dashboard variable as a production alert scope. Grafana Alerting evaluates alert rules independently of the variable values selected in a dashboard. If an engineer filters the dashboard to namespace=staging, that does not restrict which alerts fire. Alert rules must define their own labels, folders, and notification policies. Do not rely on a dashboard filter as a safety boundary.

Trade-offs and Limitations

Curated hierarchies reduce risk but add friction. A new service must exist in the expected cluster and namespace labels before it appears in the dropdown; mislabeled targets silently vanish from the variable. Regex escaping inside variable queries is also easy to get wrong—use the variable preview and a small selection before saving. Finally, variable query syntax and UI defaults differ across Grafana versions, so verify behavior in a non-production instance first.

Actionable Closing

Make variable cost part of dashboard review. Before merging a dashboard change, run the cardinality check on any new label_values variable, test with a single selection, then inspect panel query counts under All. Prefer stable label hierarchies over raw high-cardinality exports. If a variable causes load problems after deployment, remove it in Dashboard settings → Variables or restore a previous dashboard JSON version from history. The goal is not fewer dashboards; it is dashboards whose query cost stays predictable as the fleet grows.