Turning on New Relic Distributed Tracing in Java Microservices: What to Enable, How to Verify, and When to Scale

Why Enable Distributed Tracing?

When a request travels through several microservices, the latency you see in an end‑to‑end dashboard hides the true source of delay. Distributed Tracing stitches together the individual spans—each representing a method call, database query, or outbound HTTP request—into a single trace. With that view you can see which service, call, or even line of code is the bottleneck.

Getting the Java Agent Ready

The New Relic Java agent is the only component that injects trace context into your application. Install it by downloading the newrelic.jar and placing it in a directory that the JVM can access.

Agent launch command

Run your application with the agent attached. Replace /opt/newrelic/newrelic.jar and /opt/newrelic/newrelic.yml with the paths you use.

java -javaagent:/opt/newrelic/newrelic.jar \
     -Dnewrelic.config.file=/opt/newrelic/newrelic.yml \
     -jar my-service.jar

Permissions: the user that starts the JVM must have read access to the jar and the config file.

Minimal newrelic.yml snippet

# newrelic.yml
common:
  app_name: "MyJavaMicroservice"
  license_key: "YOUR_LICENSE_KEY"
  distributed_tracing.enabled: true
  # Optional: limit the amount of data collected
  transaction_tracer.enabled: true
  transaction_tracer.transaction_threshold: apdex_f
  # Sample only 10% of requests to reduce overhead
  distributed_tracing.sample_rate: 0.1

Key flag: distributed_tracing.enabled: true. When you set this, the agent starts adding traceparent headers to outbound HTTP calls and expects those headers on inbound calls.

Verifying Traces Appear in the UI

1. Send a test request to your service. Use a tool like curl or Postman.
2. Navigate to the New Relic UI → APM → Your Service → Distributed Tracing.
3. Search for the request ID (the traceId printed in the logs if you enable log.level: debug in the agent).
4. Inspect the trace tree to confirm that spans from downstream services are present.

Risk: If you don’t see any traces, double‑check that the agent is actually running (look for the New Relic Java agent banner in the application logs) and that the license key is correct.

Measuring the Performance Impact

Distributed Tracing adds byte‑code instrumentation and context propagation. In high‑traffic services, this can raise CPU usage by 3–8% and memory by a few megabytes per JVM. The exact numbers depend on:

• Number of outbound HTTP calls per request.
• Size of the trace context headers.
• Sampling rate.

Run a load test in a staging environment:

# Example using k6
import http from "k6/http";
import { check } from "k6";

export let options = {
  stages: [
    { duration: "1m", target: 100 },
    { duration: "2m", target: 100 },
    { duration: "1m", target: 0 }
  ]
};

export default function () {
  let res = http.get("https://my-service.local/api/health");
  check(res, { "status was 200": r => r.status === 200 });
}

During the test, monitor jstat or your APM’s CPU/Memory graphs. Compare the metrics with and without the distributed_tracing.enabled flag to quantify overhead.

Data Retention, Sampling, and Privacy

1. Retention: New Relic stores trace data for 30 days on the free tier. If you’re on a higher tier, retention can be extended. Exceeding the limit discards the oldest traces. Plan your sampling rate accordingly.

2. Sampling: The distributed_tracing.sample_rate defaults to 1 (100%). Setting it to 0.1 samples only 10% of requests, balancing visibility and cost.

3. Privacy: The agent can capture entire request bodies. To avoid exposing PII, use the request_body_enabled flag and the request_body_filter list:

distributed_tracing.request_body_enabled: true
distributed_tracing.request_body_filter:
  - "password"
  - "credit_card"

Without these filters, sensitive fields may appear in the UI, violating compliance policies.

When to Turn It On

Start with a subset of services or a traffic percentage. For example, enable tracing only on the service that aggregates data for the front end. Once you confirm that the overhead is acceptable and that the trace data is actionable, roll out to the rest of the stack.

Actionable Checklist

1. Deploy the Java agent with distributed_tracing.enabled: true in a staging environment.
2. Send a known request and confirm a trace appears in the New Relic UI.
3. Run a load test and compare CPU/memory usage with a baseline.
4. Adjust sample_rate and request_body_filter to meet cost and privacy requirements.
5. Gradually enable tracing in production, monitoring performance and data volume.

By following this path, you avoid surprises and ensure that Distributed Tracing delivers real insight without compromising service stability.