Diagnose Cross-Service Latency with New Relic Java Agent Tracing

The problem: slow requests disappear between services

Your dashboard shows elevated response times, but the logs in each service look normal. One service reports 800 milliseconds while its caller reports two seconds. The missing time lives in the network, a downstream database, or a message queue that neither service logs in detail. Without a shared trace identifier, you are guessing which hop is responsible.

What distributed tracing does in the New Relic Java agent

Distributed tracing assigns a trace ID to an incoming request and passes it downstream in HTTP headers or message properties. The New Relic Java agent automatically instruments common frameworks such as Spring Boot, JAX-RS, and JDBC, then records spans for each operation. A span is a timed unit of work with a start time, duration, and parent relationship. When context propagation works, New Relic displays the spans as a service map and waterfall view, so you can see which service or external call consumed the time.

Enable tracing and restart the JVM

Edit newrelic.yml on each Java service. This file usually lives beside the application JAR or in /etc/newrelic, depending on how you installed the agent.

common: &default_settings
  license_key: 
  app_name: 
  distributed_tracing:
    enabled: true
  span_events:
    enabled: true

Run this on the application host with write access to the agent configuration directory:

sudo vim /etc/newrelic/newrelic.yml

After saving, restart the JVM so the agent reloads the configuration. A restart is required because the agent instruments classes during startup.

Expected check: send a request through the instrumented service and look for the trace in New Relic APM under Distributed Tracing. If you see separate, unconnected spans per service, context propagation is not working.

Add custom spans where automatic instrumentation stops

The agent does not instrument every library or custom network layer. Use the New Relic Java API to mark business methods as spans. Example:

import com.newrelic.api.agent.Trace;

@Trace(dispatcher = true)
public Order processOrder(OrderRequest request) {
    // custom business logic
}

The @Trace annotation creates a span for the method without XML configuration. Use it sparingly; every custom span generates data and can increase agent overhead.

Trade-offs and how to verify

• Unsupported protocols or custom transport layers can break trace context, leaving disconnected spans.
• Excessive custom instrumentation raises CPU and memory usage and may increase span-event billing.
• Infinite tracing sends spans to a New Relic trace observer, which helps high-volume systems where sampling might drop the trace you need.
• Retention and sampling limits vary by account tier, so older traces may not be available for long-term analysis.

Practical verification: deploy to one service first, send a test request with a unique header such as x-test-trace: verify-123, and confirm the trace appears in New Relic with linked spans across services before rolling out to all hosts.

Rollback

Because this changes agent configuration and requires a JVM restart, you can revert by restoring the previous newrelic.yml, setting distributed_tracing.enabled to false, and restarting the JVM again.

Closing

Start with one service, verify that trace IDs survive the network boundary, then expand. Distributed tracing will not fix slow code, but it will show you exactly where to look.