Diagnosing and Reducing High Cardinality in Prometheus

Problem and Takeaway

Prometheus stores every unique label combination as a separate time series. When labels such as user_id, request_id, or timestamps acquire many distinct values, the series count explodes. The result is higher RAM usage in the TSDB head block, slower query evaluation, and eventual OOM kills. The practical takeaway: measure series cardinality first, then eliminate or aggregate the offending labels before they reach the storage layer.

Recognizable Condition

• Prometheus process restarts with out of memory in the logs.
• Query latency spikes for simple selectors (e.g., up).
• Disk usage grows faster than expected for the retention window.
• Metric prometheus_tsdb_head_series shows a steep upward trend.

Cause / Diagnostic Table

Observed Symptom	Likely Cause	Quick Check
Memory pressure / OOM	Labels with unbounded value sets (user IDs, request IDs, container IDs)	Run topk(10, count by (label) ({__name__=~".*"})) in the expression browser.
Slow range queries	Inverted index size grows with unique label pairs	Inspect prometheus_tsdb_index_size_bytes.
Disk usage outpaces retention	High‑cardinality series prevent block compaction	Compare prometheus_tsdb_head_series vs. prometheus_tsdb_compaction_retention_duration_seconds.

Ordered Diagnostic Checks

1. Total series count – Execute via HTTP API (read‑only token sufficient):

   curl -G 'http://prometheus:9090/api/v1/query' \
     --data-urlencode 'query=count({__name__=~".*"})'
   

   Expected: a single scalar. If > 5 million on a modest node, investigate further.
2. Top contributing labels – Run in the UI or API:

   topk(10, count by (label) ({__name__=~".*"}))
   

   Returns the ten labels with the highest distinct value counts.
3. Head block series metric – Scrape prometheus_tsdb_head_series over time. A steady climb indicates new high‑cardinality series are being created.
4. Index size growth – Query prometheus_tsdb_index_size_bytes. Correlate spikes with label changes.
5. Log inspection – Look for level=error msg="out of memory" or slow query warnings in Prometheus logs.

Fixes Tied to Findings

1. Remove the label at the source

If the offending label is added by instrumentation (e.g., a request_id label on HTTP metrics), modify the client library or middleware to omit it. This stops new series from being created immediately.

2. Relabel before ingestion

Use a relabel_config in prometheus.yml to drop or hash high‑cardinality labels:

scrape_configs:
  - job_name: 'myapp'
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_request_id]
        action: drop
      - source_labels: [__meta_kubernetes_pod_label_user_id]
        action: replace
        target_label: user_hash
        replacement: '{{ md5 .Value }}'
        # keeps a bounded set of hashed values

Reload Prometheus (curl -X POST http://prometheus:9090/-/reload) – requires --web.enable-lifecycle.

3. Pre‑aggregate with recording rules

Create a rule that aggregates away the high‑cardinality label:

groups:
- name: cardinality-reduction
  rules:
  - record: job:http_requests_total:rate5m
    expr: sum by (job, method, path) (rate(http_requests_total[5m]))

Queries should then use job:http_requests_total:rate5m instead of the raw metric. Note: recording rules add CPU load and do not delete existing series.

4. Delete existing series (destructive)

Only if you must reclaim space immediately and have the admin API enabled (--web.enable-admin-api):

curl -X POST -g 'http://prometheus:9090/api/v1/admin/tsdb/delete_series' \
  --data-urlencode 'match[]={__name__=~".*",request_id=~".+"}'

Risk: irreversible; disk space is not freed until the next compaction cycle or a manual POST /api/v1/admin/tsdb/clean_tombstones.

Escalation Criteria

• Series count > 10 million on a single node with < 16 GB RAM – consider sharding or remote storage (Thanos, Cortex).
• Repeated OOM despite label removal – verify that no hidden label (e.g., __tmp_ from service discovery) is still expanding.
• Compaction lag > 2× retention window – may need to increase --storage.tsdb.retention.size or add a sidecar compactor.

Limitations & Verification

Removing a label stops new series creation but does not instantly shrink the head block. The metric prometheus_tsdb_head_series will only drop after the current head block is persisted and a new one starts (default ~2 h). Verify by watching the metric for at least two head‑block cycles.

Recording rules add a new series per aggregation key; ensure the aggregated cardinality is lower than the raw metric. Use the same topk check on the recorded metric name.

Deletion via the admin API is a one‑way operation. Always snapshot the TSDB directory before running a delete request.

Quick Verification Checklist

• curl -G 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=prometheus_tsdb_head_series' – should trend down after remediation.
• curl -G 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=topk(5, count by (label) ({__name__=~".*"}))' – offending label no longer in top‑5.
• Log tail: journalctl -u prometheus -f | grep -i "out of memory\|slow query" – no new entries for 24 h.