Turn on New Relic Distributed Tracing for a Spring Boot App and Spot Latency Hotspots

Problem

In a microservice architecture a single HTTP request can hop across dozens of services. Traditional APM metrics show overall latency, but they hide where the delay actually occurs. When a request stalls, you often end up chasing logs or adding instrumentation until you find the culprit.

Why Distributed Tracing Helps

Distributed Tracing records a span for every logical unit of work – a controller method, a JDBC query, an HTTP call, etc. The spans are linked by parent‑child relationships, giving you a full picture of the request’s journey. New Relic’s UI lets you drill into each span, see its duration, and identify the exact service or operation that is the bottleneck.

Enabling Tracing in a Java Spring Boot App

New Relic’s Java agent automatically instruments Spring MVC, JDBC, and common HTTP clients. To turn tracing on you only need to edit newrelic.yml and restart the application.

1. Verify agent version: Tracing is fully supported from agent 7.0 onward. In the application root run:

   cat newrelic.yml | grep agent_version
   

   Ensure the value is 7.0 or newer.

2. Open newrelic.yml and add or update:

   distributed_tracing:
     enabled: true
     # Optional: reduce data volume
     sample_rate: 0.2
   

   The sample_rate is a float between 0 and 1; 1 means trace every request, 0.2 means 20% of requests.

3. Restart the Spring Boot service. If you’re running in Docker, rebuild the image or restart the container:

   docker compose down && docker compose up -d
   

4. Check the agent log to confirm tracing started:

   grep "DistributedTracing" newrelic_agent.log
   

   You should see an entry similar to DistributedTracing: started. If the log contains warnings about unsupported libraries, consult the Java agent docs.

Concrete Example: Tracing a REST Call

Below is a minimal Spring Boot controller that calls an external service via RestTemplate. With tracing enabled the agent will automatically create a span for the outbound HTTP request.

package com.example.demo;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

@RestController
public class DemoController {
    private final RestTemplate restTemplate = new RestTemplate();

    @GetMapping("/weather")
    public String getWeather() {
        // External call – will be traced automatically
        return restTemplate.getForObject("https://api.weather.com/v3/wx/conditions/current", String.class);
    }
}

Trigger the endpoint (e.g., curl http://localhost:8080/weather). In the New Relic UI navigate to Distributed Tracing → Traces and locate the trace for the request. You’ll see a span tree: the root span is the HTTP request to your service, a child span for the RestTemplate call, and another child span for the JDBC query if you added one.

Metrics vs Tracing: What You Gain

Metric	What It Shows	What Tracing Reveals
Average Response Time	Overall latency	Where latency is spent across services
Error Rate	Number of failures	Exact span that failed and why
Throughput	Requests per second	Which spans are hot spots under load

Trade‑Offs and Limitations

• Performance overhead: Each span adds ~1–3 ms of latency. In high‑traffic services this can be noticeable.
• Storage and cost: Traces consume more storage than plain metrics. Monitor your data usage and adjust the sample_rate if needed.
• Data sensitivity: Span attributes may include request URLs, headers, or body snippets. Mask or omit sensitive fields before sending traces to New Relic.
• Version compatibility: Agents older than 7.0 lack automatic instrumentation. Upgrade to avoid manual span creation.
• Custom libraries: If you use non‑standard HTTP clients or custom instrumentation, you may need to manually create spans via the New Relic API.

Actionable Checklist

1. Confirm agent version ≥7.0.
2. Set distributed_tracing.enabled: true and choose a sample_rate that balances insight with cost.
3. Restart the service and verify with grep DistributedTracing newrelic_agent.log.
4. Trigger a request and look for the trace in the UI.
5. Review span attributes for sensitive data and add masking if necessary.
6. Monitor data usage in the New Relic UI and adjust sampling or disable tracing on low‑impact services.

By following these steps you’ll be able to pinpoint latency hotspots in a microservice stack and make informed performance decisions without digging through endless logs.