Using New Relic Distributed Tracing to Pinpoint Latency in Microservices

Problem: Hidden Latency in a Distributed System

In a microservice stack a single user request can hop through dozens of services, databases, and external APIs. A 200‑ms delay in one hop can add up to a noticeable slowdown for the end user, but traditional metrics often hide where the delay occurs. The challenge is to see the full journey of a request and identify the exact span that is slow.

Thesis: Distributed Tracing Gives You a Single View of a Multi‑Hop Request

New Relic’s Distributed Tracing feature attaches a unique trace ID to every request and propagates it across service boundaries. Each service emits spans that represent a unit of work—like a method call or a database query. By correlating spans with the same trace ID you can map the entire request path, detect the slowest span, and focus your optimization efforts.

Setting Up Tracing Across Your Services

1. Install the Agent and Enable Tracing

New Relic agents come pre‑configured for distributed tracing, but you must enable it in the agent config. For a Java application using the Java agent, add the following to newrelic.yml or set the environment variable:

distributed_tracing:
  enabled: true
  # Optional: set a sampling rate to reduce overhead
  sampling_rate: 0.1  # 10% of requests

Run the agent with java -javaagent:/path/to/newrelic.jar -jar app.jar. Repeat the same step for each microservice (Node.js, Python, Go, etc.).

2. Propagate Headers in Custom Code

When you perform asynchronous work or call external services without the agent’s automatic instrumentation, you must manually inject the trace context headers. For example, in Java with the HttpClient:

HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("http://service-b/api"))
    .header("traceparent", TraceContext.getTraceparent())
    .build();

Missing these headers will produce broken traces where the downstream service starts a new trace instead of continuing the existing one.

Locating a Trace in the UI

After a request has completed, log into New Relic and navigate to the Distributed Tracing tab. Search for the trace_id you logged on the client side (most agents expose it in logs). The UI will show a timeline of spans, each labeled with the service name and operation.

Example: A trace might start at Gateway, go to Service A, then to Service B, and finally hit a Database. The timeline might reveal that the call to the database took 180 ms, whereas all other spans were under 20 ms.

Using NRQL to Find Slow Patterns

NRQL lets you aggregate trace data across the fleet. A common query to surface the slowest transactions is:

SELECT count(*) FROM Transaction
WHERE duration > 1.0
SINCE last 1 hour
FACET name
ORDER BY count DESC
LIMIT 10

This returns the top 10 transaction names that exceeded one second in the past hour, sorted by occurrence. You can drill down into a specific transaction and view its trace details.

Trade‑Off: Sampling vs. Visibility

Tracing every request can generate large volumes of data and increase ingestion costs. New Relic’s sampling_rate setting allows you to capture a representative subset. However, if the bottleneck occurs in a rare path, a low sampling rate might miss it. A practical approach is to start with a moderate rate (e.g., 10%) and increase it for services that handle critical, high‑volume traffic.

Also be mindful of over‑instrumentation. Adding a span for every internal method call can inflate trace size and add latency. Focus on spans that cross service boundaries or interact with external resources.

Practical Check: Verify a Trace Path

1. Send a test request to the gateway (e.g., curl -v http://gateway/api/health).
2. Log the trace_id from the response header traceparent or from the agent log.
3. In New Relic, search for that trace_id under Distributed Tracing and confirm the timeline matches the service call order.
4. Note any span that exceeds your latency SLA and investigate the underlying code or dependency.

Conclusion: From Data to Action

Distributed Tracing turns opaque request flows into visible, actionable data. By enabling tracing, ensuring proper header propagation, and querying trace data with NRQL, you can pinpoint the exact service or dependency that is slowing down your application. Balance sampling to keep costs manageable, and focus spans on critical edges. With this approach, latency bottlenecks become discoverable, measurable, and fixable—leading to a smoother user experience and more efficient microservice architecture.