Diagnosing High Error Rates in New Relic APM for Java Applications

Recognizable Condition

In New Relic APM you notice that the error rate for a specific service or transaction exceeds the threshold you have set (for example, >1% of requests) over the last 5‑minute window. The condition is visible in the APM > Errors > Inbox view, where the error count and associated stacktrace are displayed.

Cause/Diagnostic Table

Observed Signal in New Relic	Likely Root Cause
High error rate + a clear Java stacktrace (e.g., NullPointerException, SQLException)	Unhandled exception in application code
High error rate + elevated GC pause time or heap usage near max	JVM memory pressure leading to OutOfMemoryError or forced GC
High error rate + rising DB timeout errors (e.g., SQLTimeoutException)	Exhausted JDBC connection pool
High error rate + increased latency to an external service	Third‑party API failure or network issue

Ordered Checks

1. Verify error details in New Relic
   • Where: New Relic UI → APM → Errors → Inbox.
   • Permissions: User with APM view access.
   • What to check: Click the top error to see the stacktrace, frequency, and the transaction name. Note the exact exception class and method.
   • Risk: None (read‑only).
2. Correlate with JVM metrics
   • Where: New Relic Infrastructure → Hosts → select the JVM process → Memory and GC tabs.
   • Permissions: Infrastructure read access.
   • What to check: Heap usage trend, GC pause time, and GC frequency. Look for heap >80% of max or GC pauses >100 ms.
   • Risk: None (read‑only).
3. Check connection pool utilization
   • Where: If you instrument a datasource (e.g., HikariCP, Tomcat JDBC), New Relic shows JDBC metrics under APM → Databases. Otherwise, enable the New Relic Java agent’s datastore metrics or expose pool metrics via JMX and view them in Infrastructure.
   • Permissions: APM view; optionally JMX read if using custom metrics.
   • What to check: Active connections vs. max connections, idle timeout, and wait time. High active count hitting the max indicates exhaustion.
   • Risk: None (read‑only).
4. Review recent deployments or configuration changes
   • Where: New Relic Deployments page or your CI/CD system.
   • Permissions: Deployment view access.
   • What to check: Timestamp of the error spike correlates with a recent code push, library version bump, or pool size change.
   • Risk: None (read‑only).
5. Test the suspected code path locally or in staging
   • Where: Your development environment or a staging replica.
   • Permissions: Ability to run the application and attach a debugger.
   • What to check: Reproduce the request that triggered the error and verify the exception or behavior matches the stacktrace.
   • Risk: None if using a non‑production environment.

Fixes Tied to Findings

• Unhandled exception in code – Apply a targeted fix (e.g., add null checks, proper exception handling) or wrap the offending call in a try/catch block and log appropriately. Deploy the change and monitor.
• JVM memory pressure – Increase heap size (-Xmx) or switch to a GC algorithm better suited for your workload (e.g., G1GC). If the issue is a leak, use a memory profiler to locate the offending objects.
• Connection pool exhaustion – Raise the maximum pool size (e.g., HikariCP’s maximumPoolSize) or reduce the connection timeout/idle timeout to reclaim stale connections faster. Ensure the database can support the higher connection count.
• Third‑party API failure – Introduce a circuit‑breaker pattern (e.g., Resilience4j) or retry with exponential backoff. Consider fallback responses or degraded mode.

Escalation Criteria

Escalate to the platform or reliability team when:

• The error rate remains above threshold after applying the fix and verifying for two consecutive 5‑minute windows.
• JVM metrics show sustained heap >90% or GC pauses >500 ms despite heap tuning.
• Connection pool metrics indicate exhaustion even after raising the max pool size, suggesting a deeper database capacity issue.
• External service latency spikes persist and are outside your control, requiring a vendor‑level incident.

Limitations and Practical Verification

New Relic samples errors; low‑volume bursts may not appear immediately in the Errors inbox. To improve visibility, extend the time window to 15 or 30 minutes when confirming that a fix has reduced the error rate.

After a change, verify the following:

1. Error rate in APM falls below the defined threshold for at least two consecutive 5‑minute windows.
2. Related metrics (heap usage, GC pause time, connection pool utilization) return to normal baselines.
3. End‑to‑end transaction latency and throughput show no regression—compare pre‑ and post‑change charts in APM or Synthetics.

Rollback Plan (State‑Changing Operations)

If you adjust JVM heap size, GC settings, or connection pool parameters, an application restart is required. Prepare a rollback by:

• Storing the previous JVM arguments (e.g., in a version‑controlled startup script).
• Keeping the prior connection‑pool configuration file backed up.
• Having a documented restart procedure and a health‑check endpoint to confirm the service is ready before shifting traffic back.

Test the rollback in a staging environment first to ensure the process restores the previous error‑rate baseline.