Does New Relic's Java agent guarantee heap dump capture when memory usage exceeds the configured threshold in a Kubernetes pod with restricted capabilities?
0 reputation · 24 Dec 2024, 13:41 UTC
0 reputation · 24 Dec 2024, 13:41 UTC
Determine whether New Relic's Java agent guarantees heap dump capture whenever the configured memory‑usage threshold is exceeded, especially when the JVM runs inside a Kubernetes pod with limited Linux capabilities.
The agent’s automatic heap dump is described as a best‑effort feature that relies on HotSpot diagnostic commands (jmap/jcmd) and requires the process to have SYS_PTRACE or run as a privileged user. In restricted containers these commands may fail, causing the agent to skip dump generation without reporting an error. Additionally, the threshold‑based trigger can miss short‑lived or sub‑threshold leaks, and misconfiguration can suppress useful dumps.
Does the agent log a failure when it cannot execute the heap‑dump command due to missing capabilities? If a dump is not generated, does the agent send any alternative memory‑diagnostic data to New Relic? Are there any retry or fallback mechanisms that the agent employs when the initial trigger fails?
32775 reputation · 24 Dec 2024, 23:28 UTC
No. The New Relic Java agent's threshold-based heap dump is best-effort. Crossing the configured memory threshold can trigger an attempt, but it does not guarantee that a dump is captured, completed, or uploaded—especially inside a Kubernetes pod with restricted Linux capabilities.
Confirmed: The feature depends on HotSpot diagnostic mechanisms, such as attach-based jcmd/jmap flows or in-JVM diagnostic APIs. In restricted containers, dropped capabilities (for example SYS_PTRACE), non-root UIDs, seccomp profiles, read-only dump paths, disk quotas, or JVM flags that disable attach can cause the attempt to fail.
Not guaranteed: Whether the agent logs a clear failure, whether it retries, and whether it uploads alternative diagnostic data are version- and configuration-dependent. Do not assume silence means success or failure; confirm against your exact agent version and log output.
Even when a heap dump is not generated, the agent can continue to report ordinary JVM memory metrics—heap and non-heap usage, GC activity, and thread counts. These indicate memory pressure but do not provide heap-object evidence.
Retry and fallback behavior is not guaranteed. Any cooldown, re-trigger, or harvest behavior should be verified for your specific agent version and configuration. If automatic capture is business-critical, plan for a manual or sidecar-based dump path.
securityContext, capabilities, seccompProfile, UID/GID, and mounts. Set the agent log level to the most verbose supported value.jcmd/jmap, HotSpot diagnostic, file-write, permission, or disk-space errors.# Test HotSpot diagnostic dump from the same container
jcmd $(pgrep -f java) GC.heap_dump /tmp/test-heap.hprof
# Search agent logs for related errors
grep -iE 'heap.?dump|jcmd|jmap|attach|ptrace|permission|denied' <agent-log-file>
One missing detail would change the recommendation: the exact agent version and the heap-dump-related configuration names and values in use. With those, the advice can move from general verification to a specific configuration or capability adjustment.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.