Drop High‑Cardinality Labels at Scrape Time with Prometheus metric_relabel_configs

Why Cardinality Matters in Prometheus

Prometheus stores every unique combination of label names and values as a distinct time series. When an exporter emits a label with unbounded values – like user_id, request_path, or container_id – the number of series can explode. This growth translates directly into higher memory usage, disk I/O, and slower queries, and can eventually hit the max_series_per_metric limit or cause the TSDB to thrash.

Dropping Labels on Scrape with metric_relabel_configs

Instead of changing the application to stop emitting the problematic label, Prometheus gives you a way to prune it from the sample stream before it reaches the TSDB. The metric_relabel_configs section of a scrape job is evaluated for every sample, and you can use the labeldrop action to remove specific labels or drop to discard entire metrics.

Typical Use‑Case: HTTP Path Label

Suppose your node_exporter or a custom HTTP exporter emits a path label that contains the full request URI. Each unique path creates a new series for the http_requests_total metric, quickly ballooning cardinality.

Config Example

scrape_configs:
  - job_name: "http-exporter"
    static_configs:
      - targets: ["http-exporter:8080"]
    metric_relabel_configs:
      # Drop the "path" label from every sample in this job
      - action: labeldrop
        regex: "path|uri"
      # Optionally drop the entire metric if it is not needed
      #- action: drop
      #  source_labels: ["__name__"]
      #  regex: "http_request_duration_seconds"

Place this snippet in the prometheus.yml file on the machine running Prometheus. After editing, reload the configuration with a POST to /-/reload (e.g., curl -X POST http://localhost:9090/-/reload). The reload requires the prometheus user to have write permission on the configuration file and the ability to send HTTP requests to the /reload endpoint.

Regex Pitfalls

Prometheus uses RE2 syntax and anchors patterns at both ends. A pattern like path will only match the exact string path. If you want to match any label name containing the word, use path|uri or .*path.*. Remember that a non‑matching regex results in the rule doing nothing, which can be confusing when you expect a drop to happen.

Verifying the Impact

After reloading the config, confirm that the series count has decreased. Two quick checks:

1. Head Stats Page: Open /status/tsdb-head-stats in the Prometheus UI. Look at prometheus_tsdb_head_series – it reports the current number of series in the head block. Compare the value before and after the reload.
2. Metric Query: Run count by (__name__) ({...}) to list series per metric. If http_requests_total no longer shows distinct values for path, the drop worked.

Example verification command (run on the Prometheus host):

# Before reload
curl -s http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_series | jq '.data.result[0].value[1]'
# After reload
curl -s http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_series | jq '.data.result[0].value[1]'

Replace localhost with your Prometheus server address. The output will show the series count; a noticeable drop indicates success.

Trade‑Offs and Limitations

• Irreversible Loss of Dimension: Once a label is dropped, Prometheus can no longer differentiate series by that dimension. If you later need the granularity, you must re‑scrape with the original config or re‑import the data.
• Potential Series Merging: Removing a label may cause previously distinct series to merge. If the remaining label set is not unique per target, you may end up with duplicate samples or misleading aggregates.
• Version Differences: The behavior of metric_relabel_configs and the handling of native histograms changed between Prometheus 2.x and 3.x. Verify the exact semantics for your version (check prometheus --version).
• Recording Rules vs. Relabeling: Recording rules pre‑aggregate data and reduce query load, but they do not prevent raw cardinality from entering the TSDB. Use relabeling to stop the growth at the source.

When to Use Metric Relabeling

• You have a short‑lived high‑cardinality label that is useful for debugging but not for long‑term monitoring.
• Storage or query performance is becoming a bottleneck, and you need an immediate, non‑invasive fix.
• Changing the exporter is impractical due to deployment constraints or lack of source access.

Actionable Checklist

1. Identify the high‑cardinality label(s) in your metrics (use prometheus_tsdb_head_series or label_values()).
2. Draft a metric_relabel_configs rule that uses labeldrop or drop with a correctly anchored regex.
3. Deploy the rule in prometheus.yml and reload the config.
4. Verify the series count reduction and ensure queries still return expected results.
5. Document the change and monitor for any unexpected data loss or query anomalies.

By pruning labels at scrape time, you can tame Prometheus’s cardinality without touching the instrumented application. The trade‑off is a permanent loss of that dimension, so weigh the benefits against the need for future drill‑down before making the change.