Diagnosing Cross‑Service Latency in Java Microservices with New Relic Distributed Tracing

Why Distributed Tracing Matters for Microservices

When a request travels across several services, the total response time is the sum of multiple, often hidden, delays. A single slow database call, a misconfigured queue, or a network hiccup can inflate the end‑to‑end latency. New Relic’s distributed tracing gives you a single, coherent view of that journey, letting you see where the bottleneck lives.

The Concrete Problem

Imagine you have two Java services, OrderService and InventoryService. A REST call from OrderService to InventoryService is occasionally slow. You can see a high apdex in the APM dashboard, but the root cause is unclear. The question: How do I pinpoint exactly which span is responsible for the delay?

Thesis: Use Span Attributes and Service Maps Together

New Relic’s Java agent automatically propagates trace context, but to make sense of the data you need to enrich spans with meaningful attributes and then leverage the service map to correlate caller and callee latency. This blog walks through the steps, shows a minimal configuration, and discusses trade‑offs.

Step 1 – Enable Automatic Instrumentation and Custom Attributes

The agent ships with auto‑instrumentation for Spring MVC, Jersey, gRPC, and JDBC. To add a custom attribute (e.g., customer_id) you can use the NewRelic.addCustomParameter API. Below is a typical snippet you would place in a controller method:

import com.newrelic.api.agent.NewRelic;

@GetMapping("/order/{id}")
public ResponseEntity<Order> getOrder(@PathVariable("id") String orderId) {
    // Fetch order logic …
    String customerId = order.getCustomerId();
    NewRelic.addCustomParameter("customer_id", customerId);
    return ResponseEntity.ok(order);
}

Make sure the agent version is 8.x or newer; earlier releases require the legacy com.newrelic.api.agent.Trace annotation.

Step 2 – Configure Trace Sampling Rules

Tracing every request can be expensive. New Relic lets you keep high‑value traces while sampling the rest. Add the following to newrelic.yml (or the environment‑specific config file):

transaction_tracer:
  enabled: true
  transaction_threshold: apdex_f
  transaction_sampling:
    rules:
      - name: "HighValueEndpoints"
        transaction_names:
          - "OrderService/*"
        sample_rate: 1.0  # 100% for OrderService
      - name: "LowValueEndpoints"
        transaction_names:
          - "InventoryService/*"
        sample_rate: 0.1  # 10% for InventoryService

Run the agent with -javaagent:newrelic.jar and verify that the transaction_sampling rule appears in the Trace Settings page of the APM UI.

Step 3 – Verify Trace Continuity

Trigger a cross‑service request (e.g., GET /order/123). In the New Relic UI, open the Transactions tab, find the transaction, and click View distributed trace. You should see two spans: one for OrderService and one for InventoryService, both sharing the same trace ID.

To double‑check, you can query the NRQL console:

SELECT * FROM Transaction WHERE name = 'OrderService/GET /order/*' SINCE 1 minute ago

Look for the traceId field; it should match the span in InventoryService.

Step 4 – Use the Service Map

The Service Map visualizes dependencies. Open it from the APM sidebar. You should see an edge from OrderService to InventoryService annotated with average latency, error rate, and apdex. Hovering over the edge expands a breakdown of the latency components. This quick view tells you whether the caller or callee is the culprit.

Concrete Example – A Typical Scenario

Suppose OrderService calls InventoryService to check stock, but the database query inside InventoryService is slow. With the steps above, you will see:

• OrderService span: 30 ms (web + queue)
• InventoryService span: 250 ms (database 200 ms, HTTP overhead 50 ms)

The service map will highlight the edge with a high latency bar, and the trace will show the database call as the slowest sub‑span. You can now focus on optimizing the query or adding indexes.

Trade‑Offs and Limitations

• High‑cardinality attributes (e.g., user IDs) inflate storage and can slow NRQL queries. Use a hashing strategy or limit to business‑critical fields.
• Auto‑instrumentation misses custom asynchronous boundaries. If you spawn a new thread or use CompletableFuture, you must manually start a span with NewRelic.getAgent().getTracedMethod().
• Trace continuity breaks if any service in the chain lacks the agent or uses a different context propagation header (e.g., B3 vs W3C). Ensure every service is instrumented with the same agent version.

Actionable Checklist

1. Verify the Java agent is installed and auto‑instrumentation is enabled.
2. Add custom attributes where they add business value.
3. Configure sampling rules to control cost while retaining error visibility.
4. Use the service map to locate caller vs callee latency.
5. Validate trace continuity with NRQL and the distributed trace UI.
6. Monitor the impact of custom attributes on storage usage.

By following these steps, you turn opaque latency spikes into actionable insights, reduce mean time to resolution, and keep your microservices healthy.