Diagnosing and Fixing High‑Cardinality Issues in InfluxDB 2.x and 3.x

Recognizing High‑Cardinality in InfluxDB

High cardinality occurs when a measurement contains an excessive number of unique series. A series is a unique combination of tag key/value pairs. When the number of series grows into the millions, InfluxDB’s in‑memory index can consume gigabytes of RAM, slow down SELECT queries, and even trigger out‑of‑memory (OOM) crashes.

Typical symptoms:

• InfluxDB process shows memory usage > 70 % of host RAM.
• SELECT queries that filter on a tag become noticeably slower or time out.
• InfluxDB logs contain messages such as "out of memory" or "TSI index too large".
• Prometheus metrics like influxdb_mem_alloc_bytes steadily rise.

Root Causes of Excessive Series Count

Cause	Typical Tag Example	Why It Matters
High‑cardinality tags	request_id, user_ip, session_id	Each unique value creates a new series.
Timestamp‑like tags	event_time, log_timestamp	Almost every point has a unique value.
Improper tag/field separation	temperature (tag) vs. temperature (field)	Tags are indexed; fields are not.
Too many tag keys	service, host, region, env	Combinatorial explosion of series.
Retention policy mis‑configuration	long‑lived data kept in a hot bucket	Data never expires, so series accumulate.

Diagnostic Checklist

1. Confirm series count per measurement:

   influx bucket list --json | jq -r '.buckets[] | select(.name=="my_bucket") | .id'

   Replace my_bucket with your bucket name. Then query the internal _internal measurement to see series counts:

   influx query \
     'from(bucket:"_internal") \
      |> range(start: -1h) \
      |> filter(fn: (r) => r._measurement == "tsm_series_count") \
      |> filter(fn: (r) => r._field == "count") \
      |> filter(fn: (r) => r._bucket == "my_bucket")'
   

   Look for a large count value (e.g., > 5 million).

2. Check memory pressure:

   curl -s http://localhost:9999/api/v2/metrics | grep influxdb_mem_alloc_bytes

   High values relative to host RAM indicate the index is large.

3. Measure query latency:

   influx query 'from(bucket:"my_bucket") |> range(start: -5m) |> filter(fn: (r) => r._measurement == "cpu") |> keep(columns: ["_time", "usage_user"])'
   

   Run the same query with and without a tag filter. If the tag‑filtered query is significantly slower, the tag is likely part of a high‑cardinality series.

4. Inspect tag cardinality per key:

   influx query \
     'import "influxdata/influxdb/v1"
     v1.tagValues(bucket: "my_bucket", tag: "host")'
   

   Count the returned values. A count > 10 000 is a red flag.

5. Look at retention policy:

   influx bucket list --json | jq -r '.buckets[] | select(.name=="my_bucket") | .retentionRules'

   Ensure the bucket has a reasonable retention period; otherwise, data never expires.

Fixes Tied to Findings

Finding	Fix	Impact
High‑cardinality tag values (e.g., request_id)	Move the tag to a field or drop it entirely.	Reduces series count dramatically; query filtering becomes slower but acceptable for most use‑cases.
Timestamp‑like tags	Remove the tag; use the built‑in _time column instead.	Series count drops to one per measurement; query semantics unchanged.
Too many tag keys	Consolidate tags into a single composite tag or use fewer keys.	Reduces combinatorial explosion; may increase query complexity slightly.
Retention policy too long	Shorten the bucket’s retention or create a separate bucket for long‑term data.	Series count stabilizes; older data can be purged automatically.
Index not shrinking after deletion	Run a manual compaction or restart InfluxDB with --compact flag.	Reclaims memory; may cause temporary performance hit.

Concrete Example: Removing a High‑Cardinality Tag

Assume measurement http_requests contains a tag request_id that changes per request. We’ll move it to a field.

1. Create a new measurement without the tag:

   influx query \
     'from(bucket:"my_bucket") \
      |> range(start: -30d) \
      |> drop(columns: ["request_id"]) \
      |> to(bucket:"my_bucket", measurement:"http_requests_clean")'
   

2. Verify series count has dropped:

   influx query \
     'from(bucket:"my_bucket") \
      |> range(start: -1h) \
      |> filter(fn: (r) => r._measurement == "http_requests_clean") \
      |> count()'
   

3. Update retention rules or delete old data:

   influx delete \
     --start=2024-01-01T00:00:00Z \
     --stop=2024-02-01T00:00:00Z \
     --bucket=my_bucket \
     --measurement=http_requests
   

After the migration, the influxdb_mem_alloc_bytes metric should drop, and query latency on tag‑filtered queries should improve.

Escalation Criteria

• Memory usage > 80 % of host RAM for > 5 minutes.
• Query latency > 3 seconds for a 10 k‑row result set.
• InfluxDB process restarts due to OOM or internal errors.
• Prometheus alerts on influxdb_mem_alloc_bytes or influxdb_query_latency_seconds exceed thresholds.

When any of these thresholds are crossed, consider engaging your database architect or the InfluxDB support team. Provide the series count, memory metrics, and recent query logs as evidence.

Limitations and Practical Checks

• Moving tags to fields reduces query speed for tag‑based filtering. Test query performance before and after the change.
• Deleting data does not immediately shrink the index; a manual compaction or restart may be required.
• In InfluxDB 3.x (IOx), the columnar Parquet format handles high cardinality better, but query performance can still degrade with excessive tag keys.
• Always back up your bucket before performing large migrations.

To verify the fix, re‑run the diagnostic checklist. If the series count has dropped to an acceptable level and memory usage stabilizes, the issue is resolved.