Prometheus TSDB compaction and memory exhaustion during cardinality spikes

Question

Prometheus utilizes a log-structured merge-tree approach where data is written to write-ahead logs (WAL) and subsequently compacted into immutable blocks to optimize query performance.
When a system experiences a sudden spike in series cardinality, the compaction process must merge smaller blocks into larger ones. There is a technical uncertainty regarding the memory overhead required for this process when the number of unique label sets increases significantly between compaction cycles.
Given that local TSDB blocks are immutable once created, the resource demand during the merge phase can fluctuate based on the cardinality of the labels involved.

Does the compaction process have a configurable memory limit to prevent OOM events during high-cardinality spikes?
How does the TSDB prioritize block merging when memory pressure is detected?

Tasadduq Burney · Answer

Answer to the Core Questions
1. Does the compaction process have a configurable memory limit?
No – Prometheus does not expose a dedicated flag such as --storage.tsdb.compaction.memory-limit.  The amount of RAM used while compacting is driven by the number of series that can fit into the head and the concurrency of compaction jobs (default 1).  The compactor streams blocks and never loads an entire block into memory, so the peak memory is largely bounded by the series iterator overhead and the Go runtime.
2. How does TSDB prioritize block merges under memory pressure?
Compaction is scheduled based on block size and time‑range overlap.  The compactor picks the smallest blocks that overlap the same time window and merges them.  There is no dynamic reprioritization or back‑off logic that reacts to runtime memory pressure; the scheduler will continue to run until all eligible blocks are merged.
Practical Steps for Your Environment

Verify your Prometheus version and current flags:
prometheus --version
prometheus --help | grep -E 'tsdb.compaction|head-chunks|wal-compression'

Check current compaction state and block sizes:
curl -s http://localhost:9090/api/v1/admin/tsdb/blocks | jq '.data.blocks[] | {id, size_bytes, compaction_level}'

Monitor memory usage during a cardinality spike:
curl -s http://localhost:9090/api/v1/query?query=process_resident_memory_bytes | jq '.data.result[0].value[1]'
curl -s http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_series | jq '.data.result[0].value[1]'

If you see the head series count or resident memory approaching the container limit, consider:

Reducing sample_limit or label_limit in your scrape configs.
Enabling WAL compression: --storage.tsdb.wal-compression.
Increasing --storage.tsdb.retention.size or --storage.tsdb.retention.time to give compaction more time to clean up.
Running compaction with a lower concurrency: --storage.tsdb.compaction.concurrency=1 (default).

Observe the compactor logs for merge activity:
grep 'compacting blocks' /var/log/prometheus.log

Each line will list the block IDs being merged and the duration.  No memory‑pressure back‑off will appear.

What to Ask Next
If you are still seeing OOM events, it would help to know whether you are running Prometheus v3 (TSDB v3) or the legacy TSDB.  The newer engine has different memory handling and may expose additional knobs.  Also, check if a container runtime (Docker, Kubernetes) is limiting the process’s memory with cgroups – that can trigger an OOM during head ingestion even if compaction itself is fine.
Summary
Prometheus compaction does not have a dedicated memory‑limit flag, and merge order is determined by block size/time overlap, not by runtime memory pressure.  The safest way to avoid OOM during cardinality spikes is to control head size, enable WAL compression, and keep compaction concurrency low.

Prometheus TSDB compaction and memory exhaustion during cardinality spikes

1 answer

Answer to the Core Questions

Practical Steps for Your Environment

What to Ask Next

Summary

Your perspective belongs here.

0 question comments