Enable and Verify Distributed Tracing in New Relic for Microservices

Why Distributed Tracing Matters in a Microservice Stack

When a request travels through several services, the latency and errors that occur in each hop can be hard to diagnose. Distributed Tracing stitches together the individual spans (units of work) into a single trace, giving you a visual path of the request. New Relic’s agents automatically create these spans and link them with a unique Trace ID. If you want that visibility, you must make sure the trace ID is carried across service boundaries.

Getting Started: Automatic Header Injection

All New Relic agents (Java, Python, Node.js, etc.) inject the traceparent header into outgoing HTTP requests. The header follows the W3C Trace Context format:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

• Version – 00
• Trace ID – 4bf92f3577b34da6a3ce929d0e0e4736
• Parent ID – 00f067aa0ba902b7
• Trace flags – 01 (sampled)

Because the agent injects this header automatically, you usually do not need to add any code. However, you must confirm that the header is indeed present on outgoing calls and that the receiving service reads it.

Practical Verification Steps

1. Deploy two services (e.g., a Java service and a Python service) with the New Relic agent installed.
2. Trigger a cross‑service HTTP call from Service A to Service B.
3. Inspect the request headers with a debugging tool. Example using cURL:

   curl -I -X GET http://service-b/endpoint -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
   

4. Navigate to the New Relic UI → Distributed Tracing → Traces. Search for the trace ID that appears in the header. You should see a single trace that includes both Service A and Service B.
5. Check the span details to confirm that each service’s span is linked via the same trace ID.

Adding Custom Spans for Granular Insight

Sometimes the automatically generated spans are too coarse. New Relic lets you manually create spans for critical code blocks. Below is a Java example using the New Relic Java agent API:

import com.newrelic.api.agent.NewRelic;
import com.newrelic.api.agent.Trace;

public class PaymentProcessor {

    @Trace(dispatcher = true)
    public void processPayment() {
        // Start a custom span
        NewRelic.getAgent().getTracedMethod().setName("PaymentProcessing/validateCard");
        validateCard();

        // Another span
        NewRelic.getAgent().getTracedMethod().setName("PaymentProcessing/chargeCard");
        chargeCard();
    }

    private void validateCard() { /* ... */ }
    private void chargeCard() { /* ... */ }
}

Key points:

• The @Trace(dispatcher = true) annotation marks the method as an entry point for a new transaction.
• Calling setName changes the span name, making it easier to locate in the UI.
• Custom spans are only captured if the parent transaction is sampled.

Sampling: Balancing Detail and Overhead

New Relic’s sampling mechanism decides whether a transaction is fully traced. By default, it samples 1 % of all transactions, but you can adjust this in newrelic.yml or via the UI:

transaction_tracer:
  transaction_threshold: 0.5
  transaction_sample_rate: 0.05

Higher sample rates increase visibility but also memory usage and network traffic. The traceparent header’s last octet (01 for sampled, 00 for not) tells downstream services whether to record the span. If you set a very low sample rate, transient latency spikes may never appear in the UI.

Common Pitfalls and How to Avoid Them

Issue	Cause	Remedy
Broken Traces in Asynchronous Queues	Trace headers are not propagated into messages sent to Kafka or RabbitMQ.	Explicitly inject the traceparent header into the message payload or use New Relic’s OpenTelemetry exporter for the messaging system.
Excessive Span Creation	Creating a span for every loop iteration or trivial method.	Limit custom spans to business‑critical sections. Use the agent’s setIgnore flag for noisy methods.
Missing Trace IDs in External Calls	Legacy libraries that strip headers or use non‑HTTP transports.	Wrap the call with a manual span and manually add the traceparent header using the library’s request builder.
Sampling Skips Long‑Tail Latencies	Sampling window too aggressive (e.g., 0.01).	Increase transaction_sample_rate or enable long‑tail sampling if available.

Integrating with OpenTelemetry

If your services use OpenTelemetry (OTel) instead of native New Relic agents, you can still send traces to New Relic by configuring the New Relic OTel exporter. The exporter automatically translates OTel spans into New Relic traces, preserving trace IDs and baggage. Example configuration for a Python OTel application:

from opentelemetry.exporter.newrelic import NewRelicSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
exporter = NewRelicSpanExporter(api_key="YOUR_NEW_RELIC_KEY")
provider.add_span_processor(BatchSpanProcessor(exporter))

# Set the provider globally
from opentelemetry import trace
trace.set_tracer_provider(provider)

When using OTel, ensure that the traceparent header is propagated by the HTTP client library (most modern clients do this automatically).

Practical Checklist Before Going Live

• Verify that every outbound HTTP client call contains the traceparent header.
• Confirm that asynchronous message producers inject the header into the message payload.
• Review the sampling configuration to match your performance and cost goals.
• Test custom span creation in a staging environment; monitor memory usage and agent logs for throttling warnings.
• Use the New Relic UI’s trace explorer to confirm that a single trace ID appears across all services.

Conclusion

Distributed Tracing in New Relic is powerful but requires careful setup. Automatic header injection handles most cases, but you must still be vigilant about async systems, sampling rates, and custom instrumentation. By following the verification steps above and keeping the checklist in mind, you’ll gain reliable end‑to‑end visibility into your microservice architecture.