glibc malloc arena retention vs. MALLOC_ARENA_MAX configuration

Memory Retention in Multithreaded glibc Environments

In multithreaded Unix applications using glibc, the allocator creates per-thread arenas to reduce lock contention. By default, glibc may allocate up to eight arenas per CPU core on 64-bit systems. This behavior often results in Resident Set Size (RSS) growth that mimics a memory leak, as freed memory remains cached within these arenas rather than being returned to the operating system.

While the MALLOC_ARENA_MAX environment variable can cap the number of arenas to reduce the memory footprint, this introduces a trade-off between memory efficiency and synchronization overhead. There is uncertainty regarding the optimal balance when scaling across high-core-count systems where default arena counts may lead to excessive virtual memory mapping.

How can an operator distinguish between genuine heap growth and arena-based retention using mallinfo2() or malloc_stats()? What is the expected impact on p99 latency when restricting MALLOC_ARENA_MAX in a highly concurrent workload?