New Relic Distributed Tracing: End-to-End Visibility for Microservices

Problem: Latency hides in the call chain

When a request travels through several services, a slowdown in one downstream component can appear as a vague increase in overall response time. Without a way to see the exact path and timing of each hop, engineers spend time guessing which service is responsible.

Thesis: New Relic's distributed tracing gives automatic, end-to-end visibility

By instrumenting each service with the New Relic agent, a unique trace ID is injected into the incoming request and propagated via standard W3C TraceContext headers. Every hop creates a span that records service name, operation type, HTTP status, and any custom attributes you add. The collected traces can be queried with NRQL and visualized as flame graphs or service maps, letting you isolate latency outliers or error spikes quickly.

How automatic trace propagation works

The New Relic agent for your language (e.g., Node.js, Java, .NET) reads the incoming request for a traceparent header. If none is present, it generates a new trace ID and adds the header to the outgoing request. Downstream services that also run the agent read that header and continue the same trace ID, creating an unbroken chain without manual code changes.

Span attributes and custom enrichment

Each span automatically captures:

• Service name (from the agent configuration)
• Operation type (e.g., HTTP request, database query)
• HTTP status code and method
• Duration of the operation

You can add custom attributes via the agent API, for example:

// Node.js example
newrelic.addCustomAttribute('userId', request.user.id);
newrelic.addCustomAttribute('featureFlag', 'newCheckout');

These attributes become searchable fields in NRQL and appear in the trace details view.

Querying and visualizing traces

In New Relic One, navigate to APM → Distributed tracing to see a list of recent traces. Selecting a trace shows a flame graph where each block represents a span; hovering reveals duration, status, and attributes.

For programmatic analysis, you can run NRQL against the Trace or Transaction events. A simple query to fetch recent trace IDs and durations looks like:

SELECT traceId, duration FROM Transaction WHERE appName = 'frontend-service' SINCE 10 minutes ago LIMIT 5

This returns rows where each traceId corresponds to a full distributed trace, and duration is the total request time.

Worked example: tracing a request from frontend to backend

Assume you have two Node.js services: frontend-api (receives HTTP calls) and backend-worker (called by the frontend). Both have the New Relic agent installed.

1. Configure the agent – Ensure newrelic.js contains your license key and distributed_tracing.enabled: true (default). No restart is needed if you use the agent's hot-swap feature; otherwise restart the processes.
2. Generate traffic – From a terminal or curl, call the frontend endpoint that triggers the backend call:

curl -X GET http://frontend-api.example.com/orders/123

The frontend agent adds a traceparent header to the outgoing request to backend-worker. The backend agent reads that header and continues the same trace.

Verify in the UI – In New Relic One, open APM → Distributed tracing, filter by frontend-api, and select the most recent trace. You should see two spans: one labeled frontend-api (HTTP GET) and another labeled backend-worker (the downstream call). Each span shows its service name, HTTP status, and any custom attributes you added.

Optional NRQL check – Run the following query to confirm trace IDs are being stored:

SELECT traceId, duration FROM Transaction WHERE appName IN ('frontend-api', 'backend-worker') SINCE 5 minutes ago LIMIT 10

If rows appear with matching traceId values across both services, the distributed trace is being ingested correctly.

Trade-offs and limitations

While automatic tracing reduces instrumentation effort, consider these factors:

• Data volume cost – Every request creates trace segments. High-traffic services can increase ingestion fees. Mitigate by enabling sampling (distributed_tracing.sampling.enabled: true) or adjusting the trace_observability_threshold to ignore very fast transactions.
• Propagation dependencies – The agent relies on frameworks that forward W3C TraceContext headers (traceparent, tracestate). Custom transports (e.g., raw TCP, gRPC without HTTP headers) or non-HTTP protocols may need manual header injection via the agent API.
• Retention limits – Raw trace segments are retained based on your account tier; older traces may be rolled up into summary tables, limiting deep-dive historical analysis. Check your tier's retention policy in the account settings.

Actionable closing

To start using New Relic distributed tracing today:

1. Install or update the New Relic agent for each service and verify the license key is set.
2. Ensure distributed_tracing.enabled is true (default).
3. Restart the services if required.
4. Generate a request that crosses at least two instrumented services.
5. In the UI, confirm the trace ID appears in each service's span list and that expected attributes are present.
6. Optionally run an NRQL query to validate ingestion.
7. Monitor trace volume; adjust sampling or thresholds if costs rise.

With these steps, you gain a concrete, queryable view of request paths across your microservices, turning vague latency complaints into precise, actionable insights.