Diagnosing and Fixing InfluxDB OOM Kills: A Step-by-Step Guide

Recognizable Condition

The influxd process is terminated by the Linux OOM killer. Check dmesg -T | grep -i 'out of memory' for entries like Out of memory: Kill process 1234 (influxd). Alternatively, the process RSS grows until host swap thrashing occurs, often coinciding with high cardinality series or large GROUP BY queries.

Cause and Diagnostic Table

Symptom	Likely Cause	Key Metric to Check
RSS climbs steadily	Unbounded series cardinality	SHOW SERIES CARDINALITY
Query spikes memory	Large in-memory result sets	Query log bytesRead
High cache, low load	WAL/compaction buffers not releasing	cacheMemoryBytes, walMemoryBytes
Memory jump after restart	Many shard indexes	indexMemoryBytes per shard

Ordered Investigation Steps

1. Check series cardinality. Run on the InfluxDB host (requires network access to HTTP API, typically port 8086):

   curl -G 'http://localhost:8086/query?db=mydb' --data-urlencode 'q=SHOW SERIES CARDINALITY'

   Replace mydb with your database name. Expect a numeric result; values above 10 million indicate cardinality pressure.
2. Inspect memory breakdown. For v1.x use /debug/vars; for v2.x use the Prometheus /metrics endpoint:

   # v1.x
   curl http://localhost:8086/debug/vars | jq '.cacheMemoryBytes, .walMemoryBytes, .indexMemoryBytes'
   
   # v2.x
   curl http://localhost:8086/metrics | grep influxdb_memory

   Requires jq installed. Look for cacheMemoryBytes exceeding configured cache-max-memory-size or indexMemoryBytes growing with shard count.
3. Correlate query spikes. Enable query logging in influxdb.conf:

   [query]
     query-log-enabled = true

   Restart influxd (requires service restart permissions). Then tail the log and match long-running queries with memory spikes.
4. Verify cardinality limits. Check current config:

   influxd config | grep -A2 -E 'max-values-per-tag|max-series-per-database|cache-max-memory-size'

   Ensure max-values-per-tag is not 0 (unlimited). A safe starting point is 100000 per tag.
5. Review shard group duration. Short durations (e.g., 1 day) create many shards, each with index overhead. Check retention policy:

   curl -G 'http://localhost:8086/query' --data-urlencode 'q=SHOW RETENTION POLICIES ON mydb'

   Look at shardGroupDuration. Consider increasing to 7d or 30d for lower cardinality workloads.

Targeted Fixes

1. Cardinality > 10M series

• Set max-values-per-tag = 100000 in [data] (v1) or [storage] (v2) to reject writes that exceed the limit.
• Move high-cardinality tags to fields (fields are not indexed).
• If cardinality exceeds 50M on OSS, plan migration to InfluxDB IOx (v3) or TimescaleDB.

2. Query-induced spikes

• Add LIMIT and OFFSET to large queries.
• Use GROUP BY time() with aggregates instead of raw points.
• Set max-select-point and max-select-series in config to cap result sets.

3. Cache/WAL pressure

• Raise cache-max-memory-size and cache-snapshot-memory-size only if RAM headroom exists (e.g., keep total process RSS below 70% of physical memory).
• Avoid wal-fsync-delay = 0; use a small delay (e.g., 10ms) to batch writes.

4. Shard explosion

• Increase shard-group-duration (e.g., 7d → 30d) in retention policy via ALTER RETENTION POLICY.
• Run offline index rebuild to compact TSI1 indexes: influxd inspect buildtsi -database mydb -retention autogen (requires downtime).

Escalation Criteria

Escalate to architecture change when:

1. OOM kills persist after applying config limits and schema changes for two full retention cycles.
2. Series cardinality exceeds 50M on OSS (hard instability limit ~100M).
3. Query latency P99 > 30s despite limits, indicating storage engine bottleneck.
4. Disk I/O wait > 40% during compaction — consider migrating to InfluxDB IOx (v3) or TimescaleDB.

Version-Sensitive Behavior

• v1.x uses TSM/TSI1; config keys under [data].
• v2.x uses same engine with config under [storage]; metrics via /metrics (Prometheus).
• IOx (v3) uses Apache Arrow/Parquet, handles cardinality differently, and is not API-compatible with v1/v2 line protocol authentication. Migration changes diagnostics entirely.

Limitations and Verification

This guide assumes a single-node OSS deployment with swap enabled (which masks OOM until thrashing). Enterprise features like downsampling via continuous queries (v1) or tasks (v2) run as separate queries and count against max-concurrent-queries.

Practical verification: After applying fixes, simulate a write load with 1M unique tag values and watch RSS:

pidstat -r -p $(pidof influxd) 1

Expect RSS to stabilize below the configured cache-max-memory-size plus index overhead. Confirm no OOM entries in dmesg for at least two retention cycles.

Risk note: Increasing cache-max-memory-size without RAM headroom shifts OOM from cache to process. Never set max-values-per-tag = 0 in production.