Diagnosing Data Gaps and Reporting Delays in New Relic Java APM

The Problem: Missing Spans and Delayed Metrics

When New Relic APM shows gaps in time-series graphs or reports a "Disconnected" status for a Java application, the issue is rarely a failure of the application itself. Instead, it is typically a breakdown in the telemetry pipeline—the process where the agent harvests data, buffers it in memory, and transmits it via TLS to the New Relic collector.

The primary takeaway: Most data gaps are caused by network egress blocks or JVM resource starvation, not agent crashes.

Quick Diagnostic Reference

Symptom	Likely Cause	Primary Diagnostic Indicator
Complete data blackout	Invalid License Key or Firewall Block	403 Forbidden or Connection refused in logs
Intermittent gaps / "Jagged" lines	Agent Thread Starvation	High Host CPU utilization (>90%)
Delayed reporting / Late spans	JVM Heap Pressure	Frequent Full GC events in JVM logs
Partial data (some metrics, no traces)	Buffer Overflow	Dropped spans messages in newrelic_agent.log

Step-by-Step Resolution Path

1. Verify Collector Connectivity

The agent must communicate over port 443 to the regional New Relic collector. If the host is behind a strict firewall, the agent will fail to handshake.

Run this command from the application server (replace collector.newrelic.com with your specific regional endpoint):

# Run as a user with network permissions
curl -v https://collector.newrelic.com

Expected Result: A successful TLS handshake. If the command hangs or returns a "Connection timed out," you must whitelist outbound traffic to the New Relic IP ranges on port 443.

2. Validate Authentication and Configuration

An incorrect license key will result in the collector rejecting the data stream, though the agent may appear to be running normally within the JVM.

• Open newrelic.yml (usually located in the /newrelic directory of the app).
• Verify the license_key: value matches the key provided in your New Relic account settings.
• Check the newrelic_agent.log for 403 Forbidden errors.

3. Analyze JVM Resource Contention

The New Relic agent runs as a set of background threads within your JVM. If the JVM is under extreme memory pressure or the CPU is saturated, the agent's harvester threads may be starved, leading to delayed reporting.

Check for Heap Exhaustion: If the JVM is spending significant time in Garbage Collection (GC), the agent's internal buffers may overflow, causing it to drop spans to protect the application's stability.

Check for CPU Starvation: Use top or htop on the host. If CPU usage is consistently at 100%, the agent cannot reliably package and send data on its scheduled intervals.

4. Inspect Agent Logs for Buffer Overflows

If connectivity is fine but data is still missing, check the newrelic_agent.log for warnings regarding buffer capacity. This happens when the application generates more telemetry than the agent can transmit.

Risk Warning: Do not enable log_level: debug in production without a log rotation policy in place, as this can rapidly consume disk space.

Comparison: Network Block vs. Resource Starvation

Feature	Network/Auth Block	Resource Starvation
Log Pattern	Connection refused / 403	Dropped spans / Harvest delayed
UI Status	Disconnected	Connected (but with gaps)
Recovery	Immediate after firewall change	Gradual as load decreases

Verification and Rollback

To verify the fix, monitor the APM > Agents tab in the New Relic UI. The entity should transition from "Disconnected" to "Connected," and the time-series charts should show a continuous line without gaps.

Rollback: If you modified newrelic.yml, revert the changes to the previous known-working version and restart the JVM to apply the configuration.

When to Escalate

Contact New Relic support if:

• Connectivity tests (curl) are successful and the license key is verified, but the agent still reports 403 Forbidden.
• The agent is causing significant JVM overhead (CPU/Memory) despite low application traffic.
• Logs indicate a ClassCastException or other internal agent crashes following a JVM version upgrade.