glibc malloc arena retention vs. MALLOC_ARENA_MAX configuration

Answer

1. Run the target workload under normal conditions and, at a quiet point, invoke mallinfo2() and malloc_stats() (or malloc_info(0,NULL)) from within the program or via a debugging hook.
2. Record mallinfo2.arena (bytes obtained from the system for the main arena) and the total memory reported by malloc_stats(), which sums across all arenas.
3. If the total from malloc_stats() is substantially larger than mallinfo2.arena (e.g., more than twice), the excess corresponds to memory retained in non‑main arenas.
4. Optionally, compute retained = malloc_stats_total - mallinfo2.arena and compare it to the process RSS from /proc/self/statm or pmap to see how much of the RSS is arena‑cached.

Confirmed facts

• glibc creates per‑thread arenas up to the limit set by MALLOC_ARENA_MAX; the default is roughly two times the number of CPU cores on 64‑bit builds.
• Memory that is freed to an arena stays cached inside that arena and is not returned to the OS unless the arena is destroyed or its top chunk is released via sbrk/mmap.
• mallinfo2() reports statistics for the main arena only, while malloc_stats() aggregates over all arenas.

Likely explanation

In a multithreaded program that allocates and frees many small objects, each thread may obtain its own arena. The freed memory remains in the arena’s caches, causing the resident set size to grow without a true leak. Lowering MALLOC_ARENA_MAX forces threads to share fewer arenas, which reduces the cached memory but increases lock contention on the remaining arena locks.

Expected p99 latency impact

When the workload is allocation‑intensive and highly concurrent, restricting MALLOC_ARENA_MAX can raise the 99th‑percentile latency by roughly 10‑50 % (or more) because threads spend more time waiting for arena locks. In allocation‑light workloads the effect is usually negligible.

Missing diagnostic detail that could change the recommendation

To decide whether limiting arenas is worthwhile, we need to know the average allocation size and allocation rate per thread (or the thread‑to‑core ratio). If allocations are infrequent or large, the latency penalty may be small; if they are frequent and small, the penalty could be unacceptable.