Control Prometheus Cardinality with metric_relabel_configs Before Storage

A single high-cardinality label like path or user_id can turn a healthy Prometheus instance into an expensive one. The practical engineering decision is to drop, rename or rewrite labels before samples are ingested, using metric_relabel_configs inside scrape_config. Relabeling is destructive and runs before storage, so it reduces series count, disk and query cost without changing the exporter.

Where relabeling applies in the scrape pipeline

Prometheus applies two relabel stages per job, in order.

relabel_configs runs during service discovery, before the scrape is made. It can rewrite __address__, add static labels, or drop targets entirely. It does not see metric names or metric labels.

metric_relabel_configs runs after the scrape response is parsed and before the samples are written to TSDB. It sees __name__ and all metric labels. This is the stage to use for cardinality hygiene: keep only needed metrics and strip high-cardinality labels.

Both stages are lists of rules applied sequentially to each sample. A rule is matched via source_labels, a regular expression, and an action. The order matters and mismatches are silent by default.

Worked configuration for dropping a high-cardinality label

The following pattern is a common, safe starting point for an HTTP exporter that emits http_request_duration_seconds with a path label that explodes cardinality.

scrape_configs:
  - job_name: 'app-http'
    static_configs:
      - targets: ['app:8080']
    metric_relabel_configs:
      # Keep only the metric we care about for this job
      - source_labels: ['__name__']
        regex: 'http_request_duration_seconds'
        action: keep

      # Remove the high-cardinality path label from all kept samples
      - regex: 'path'
        action: labeldrop

      # Optional: normalize instance label from __address__
      - source_labels: ['__address__']
        target_label: 'instance'
        regex: '(.*):.*'
        replacement: '${1}'
        action: replace

Place this block under the specific job you want to control. Do not apply a broad keep rule at the global level unless you intend to drop metrics for all jobs.

Why metric_relabel_configs and not relabel_configs

relabel_configs cannot see metric labels, only target labels like __meta_* and __address__. If you try to drop path there, nothing happens because the label does not exist on the target.

metric_relabel_configs sees the full sample after scrape. It can therefore filter by __name__ and manipulate metric labels. It cannot reduce the amount of data transferred from the exporter, only what is stored and queried.

Limits and common mistakes

Relabeling is destructive. Once a label is dropped in metric_relabel_configs it cannot be recovered for that scrape. Test changes in a staging scrape job first.

Order matters. A keep rule followed by a labeldrop will drop the label from the kept set. Reversing the order can keep samples you intended to discard. Rules are evaluated top to bottom.

Errors are silent. An incorrect regex will simply not match, leaving labels in place. There is no warning in the default scrape logs for a non-matching relabel rule.

metric_relabel_configs runs after scrape but before storage. It does not reduce scrape load or network transfer. The exporter still emits the high-cardinality label, and Prometheus still fetches it. The savings are in storage and query cost.

Version behavior for actions and source_labels semantics is stable across recent Prometheus releases, but exact defaults for scrape config can vary. Validate the config against your deployment.

Practical verification

Validate syntax without restarting production. Run the check on a host with read access to the config file and the Prometheus binary.

prometheus --config.file=prometheus.yml --check-config

This command requires permission to read the config file. It exits with a non-zero status on syntax errors. Risk is low, it does not reload or restart the server.

After applying a change, inspect the Prometheus Targets page for the job. Compare the active labels shown for a target before and after the change. The path label should be absent from the sample preview.

Confirm cardinality reduction with PromQL. Example queries to run in the Prometheus UI:

• count by (__name__) (http_request_duration_seconds) to see series per metric.
• count({__name__="http_request_duration_seconds"}) to see total series for the metric.

Compare the count before and after the relabel change. Expected outcome is a lower series count and the absence of the dropped label in query results.

Monitor storage and query latency metrics over a representative period. A reduction in series creation rate and more stable query latency indicates the relabel is having the intended practical effect.

If you need to revert, remove or comment the offending metric_relabel_configs block and reload the configuration. Because relabeling is applied at scrape time, new scrapes will immediately re-ingest the previously dropped labels.