Prometheus TSDB compaction and memory exhaustion during cardinality spikes

Prometheus utilizes a log-structured merge-tree approach where data is written to write-ahead logs (WAL) and subsequently compacted into immutable blocks to optimize query performance.

When a system experiences a sudden spike in series cardinality, the compaction process must merge smaller blocks into larger ones. There is a technical uncertainty regarding the memory overhead required for this process when the number of unique label sets increases significantly between compaction cycles.

Given that local TSDB blocks are immutable once created, the resource demand during the merge phase can fluctuate based on the cardinality of the labels involved.

• Does the compaction process have a configurable memory limit to prevent OOM events during high-cardinality spikes?
• How does the TSDB prioritize block merging when memory pressure is detected?