Native memory leak in Java thread‑sampling sensor causing RSS growth in Dynatrace OneAgent
0 reputation · 01 Feb 2022, 21:10 UTC
0 reputation · 01 Feb 2022, 21:10 UTC
The goal is to ascertain whether the unresolved design decision around process‑group merging in Dynatrace OneAgent could leave native thread‑sampling buffers allocated after the leak fix introduced in version 1.225+.
Observed RSS growth is tied to Java thread counts and only appears in releases prior to 1.225, but the fix notes do not detail the native buffer release path or how merging decisions interact with the sensor, creating uncertainty about residual memory impact in newer agents.
Does the process‑group merging algorithm retain references to native thread‑sampling buffers after a merge operation?
Can enabling OneAgent debug‑level thread‑sampling logging show whether those buffers are freed per sampling interval in versions 1.225+?
30775 reputation · 02 Feb 2022, 08:56 UTC
There is no public evidence that the process-group merging algorithm in OneAgent 1.225+ retains references to native thread-sampling buffers. The 1.225 fix addressed the buffer deallocation path that caused RSS growth proportional to Java thread count in earlier releases, and the pre-1.225 symptom — steady native growth tied to thread counts — is the signature of buffers never being freed at all, not of references surviving a merge. However, the release notes do not describe how merging interacts with sensor-native state, so the merge edge case cannot be confirmed from documentation alone. It has to be verified empirically on your deployment.
Established from your own observations: RSS growth correlates with Java thread counts and appears only on agents older than 1.225. That is consistent with a per-thread or per-sampling-interval native allocation that was never released, and with that path being fixed in 1.225+.
Assumed, not confirmed: that a process-group merge could orphan sensor state. A merge is primarily a grouping/identity decision in the monitoring model; whether it tears down and re-creates the Java sensor's native context is an implementation detail Dynatrace has not documented. If a merge did leak, the expected signature would be a step increase in native RSS at each merge event — not the continuous, thread-count-proportional slope you saw pre-1.225. That distinction is your best diagnostic lever.
Yes — enabling debug-level logging for the Java thread-sampling sensor is the right approach, and it is exactly how you would answer the retention question on 1.225+. The goal is to observe allocation/free pairs per sampling interval and check whether a merge event interrupts the pairing.
Logging alone shows intent, not bytes. Pair it with native memory measurement around the merge event:
# Baseline native RSS of the OneAgent Java module process\nps -o pid,rss,vsz,comm -p <pid>\n\n# Detailed native mappings (look for anonymous heap growth)\ncat /proc/<pid>/smaps_rollup | grep -E 'Rss|Anonymous'\n\n# Repeat after a controlled merge event and compare\nIf RSS stays flat across a merge while logs show matched alloc/free pairs, the merge path is clean. If RSS steps up at each merge while frees stop appearing for pre-merge allocations, you have reproduced a residual leak — at which point the log excerpts and smaps snapshots are exactly what Dynatrace support needs.
Debug logging itself allocates and can skew RSS upward; measure the baseline with logging on, not against a logging-off baseline. Behavior may also differ by OS, Java vendor, and injection mode, so a clean result on one host does not fully clear the edge case. One detail would sharpen this: are you seeing any measurable RSS growth on 1.225+ today, or is this a pre-upgrade risk assessment? If growth is currently observable, skip straight to the merge-correlated smaps comparison; if not, the debug-log check during a single controlled merge is sufficient assurance.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.