Unexplained memory growth on Heroku dyno during sustained load
0 reputation · 29 Sept 2021, 09:03 UTC
0 reputation · 29 Sept 2021, 09:03 UTC
Goal: determine whether a steady rise in RSS reported by Heroku’s dyno metrics reflects a genuine memory leak in the application or expected platform behavior such as garbage‑collection jitter or undocumented swap usage.
Constraints: Heroku exposes only the total memory footprint via ps:info, with no per‑process or heap breakdown; dynos may restart after deploys or resource rebalancing, masking gradual trends; the log drain’s 10 000‑line/minute limit can drop diagnostic output, and the exact swap allocation granularity per container is not documented, making attribution difficult without external add‑ons.
Questions: What magnitude of memory increase over multiple dyno cycles qualifies as a leak versus normal variance? How can one isolate application‑specific memory from platform overhead using only Heroku‑provided metrics? Is there a way to observe memory behavior reliably without triggering dyno restarts that would reset the measurement?
35275 reputation · 29 Sept 2021, 12:38 UTC
There is no fixed megabyte threshold that separates a leak from normal variance on Heroku — the signal is the trend, not the absolute number. A genuine leak shows monotonic RSS growth per unit of work that persists through idle periods and across multiple dyno cycles, eventually approaching your quota (and R14 errors). Normal variance is bounded jitter around a stable baseline, with periodic drops after garbage collection and a plateau once caches and runtime structures warm up.
Confirmed: Heroku reports only total container RSS via heroku ps:info and log-runtime-metrics. That number blends your application, the language runtime, OS page cache, and any other processes in the container. Swap allocation granularity per container is undocumented, so stable RSS can mask swap use.
Likely explanation for what you're seeing: sustained load inflates RSS through GC heap growth, JIT/runtime warmup, connection pools, and in-memory caches. Runtimes like the JVM, V8, and Ruby's allocator often grow toward available memory and resist returning it to the OS. This looks like a leak for hours, then plateaus. A real leak never plateaus.
Full separation isn't possible with platform metrics alone. The practical proxy:
heroku ps:info every few minutes for several hours under representative load, and plot RSS over time, annotating deploys and restarts so resets aren't mistaken for drops.Avoid deploys, scaling changes, and manual restarts during the measurement window, since each resets RSS. Enable log-runtime-metrics and emit a low-rate in-app memory sample (e.g., process.memoryUsage() or equivalent) to stdout every 30–60 seconds — well under the 10,000-line/minute log drain limit. That gives you an app-only heap/RSS series to compare against the container total: if app heap is flat while container RSS climbs, suspect runtime/allocator behavior or swap rather than your code.
Platform behavior can change, and swap accounting is undocumented — treat the baseline comparison as an estimate, and re-verify against current Heroku documentation if the numbers look anomalous. The one detail that would sharpen this diagnosis: does RSS ever plateau during a sustained quiet period, or does it rise until the dyno hits its quota?
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.