Unbounded memory growth in MeterListener when using high‑cardinality labels
0 reputation · 20 Oct 2024, 18:26 UTC
0 reputation · 20 Oct 2024, 18:26 UTC
Goal: establish a practical upper bound for the number of distinct label values that a MeterListener can safely handle before its internal storage grows without bound.
Constraints: the guidance applies to .NET 7 and later when using System.Diagnostics.Metrics with a MeterListener that collects histograms or counters; the runtime does not currently document a cardinality limit, and enabling high‑frequency collection may exacerbate memory pressure. Uncertainty remains whether the MeterListener automatically aggregates or discards excess label combinations, or if developers must implement their own sampling.
What label‑cardinality threshold triggers noticeable memory growth in a MeterListener session? Are there built‑in mechanisms to limit or aggregate high‑dimensional label sets? How can developers instrument high‑cardinality metrics safely without risking unbounded memory consumption?
35275 reputation · 20 Oct 2024, 21:58 UTC
There is no fixed cardinality threshold at which MeterListener starts consuming unbounded memory, because the listener itself does not cache or aggregate your measurements. Memory growth comes almost entirely from what your callback does with each unique tag combination. If your OnMeasurementRecorded handler stores every distinct TagList in a dictionary, memory grows linearly with the number of unique label values — there is no built-in cap, no automatic aggregation, and no eviction in System.Diagnostics.Metrics on .NET 7 or later.
What is well established:
Counter.Add, Histogram.Record, etc. is delivered to the listener as a distinct measurement context. The runtime does not deduplicate or discard excess combinations for you.TagList allocations, even if you store nothing.What remains uncertain and should be verified in your environment: whether any memory retention you observe comes from the listener pipeline itself or purely from your aggregation code. In practice, heap dumps in reported cases point at the developer's own storage collection, but treat this as a hypothesis until you dump your process.
Since no runtime limit exists, the safe bound is one you choose. A common working rule is to cap tracked tag combinations per instrument at a few thousand (for example 1,000–5,000), because each combination typically costs tens to hundreds of bytes in a dictionary plus per-histogram bucket state. Beyond that, aggregate the overflow into a single "other" bucket rather than dropping it silently.
listener.OnMeasurementRecorded = (instrument, value, tags, state) =>
{
var key = tags.ToString(); // or a cheaper structural key
if (!store.TryGetValue(key, out var entry))
{
if (store.Count >= MaxCombinations)
key = "__overflow__";
entry = store.GetOrAdd(key, _ => new AggregationEntry());
}
entry.Record(value);
};Record, so the cardinality never reaches the listener.Confirm the diagnosis and the fix with two checks:
dotnet-counters monitor --process-id <pid> and watch GC Heap Size and Gen 2 collections while recording metrics with deliberately unique tag values. Unbounded growth in the heap that tracks your unique-value rate confirms callback storage is the culprit.dotnet-dump collect -p <pid> and inspect the dominant retained objects. If your dictionary or histogram-bucket types dominate, the retention is in your aggregation code, not the listener.This guidance assumes .NET 7+ with System.Diagnostics.Metrics and a custom MeterListener. If you export via OpenTelemetry instead, the SDK's aggregation and cardinality handling differ and should be checked against its current documentation. If your heap dump shows retention inside runtime types rather than your own collections, that would change the recommendation — share the dump's dominant-type list before tuning further.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.