Hunting Latency Spikes in Kubernetes with New Relic Distributed Tracing

The 'Ghost' Latency Problem

In a Kubernetes-based microservices architecture, a slow response at the API Gateway rarely tells the whole story. When a checkout process takes five seconds instead of five hundred milliseconds, the problem could be anywhere: a saturated database connection in the inventory service, a cold start in a serverless function, or a third-party payment gateway timing out. Without distributed tracing, engineers are forced to manually correlate timestamps across multiple log streams.

The solution is to move from aggregate metrics like average response time to Distributed Tracing. By propagating a unique trace ID across every service boundary, you can visualize the exact path of a single request and identify the specific span, a timed operation, causing the bottleneck.

Implementing Trace Propagation

To get a complete picture, every service in the request chain must be instrumented. New Relic APM agents handle the heavy lifting by using the W3C TraceContext standard, which injects tracing headers into outgoing HTTP requests.

For a Java or Node.js application running in Kubernetes, ensure the agent is configured with distributed tracing enabled. While often on by default in newer versions, you can explicitly verify this in your environment variables or configuration file:

# Example environment variable for a Kubernetes Deployment spec
- name: NEW_RELIC_DISTRIBUTED_TRACING_ENABLED
  value: "true"

Critical requirement: your ingress controllers and service meshes must be configured to forward these headers. If a load balancer or proxy strips the traceparent header, the trace continuity breaks, and New Relic will treat the downstream request as a brand-new transaction rather than a continuation of the original call.

Analyzing the Flame Graph

Once data is flowing, navigate to APM > Distributed Tracing. Instead of looking at a line chart, you will see a flame graph. Each horizontal bar represents a span. The wider the bar, the longer the operation took.

• Service Spans: show the time spent within your application code.
• Database Spans: highlight slow queries or connection pooling delays.
• External Spans: track calls to third-party APIs.

Worked Example: The Order Flow Bottleneck

Consider a standard order flow: API Gateway → Inventory Service → Payment Service. A latency spike is detected, and the Distributed Tracing UI reveals the following sequence:

The flame graph makes it immediately obvious that the Payment Service is waiting 650ms for an external vendor. By analyzing the trace, the team discovered the external API was struggling with a specific region. By implementing a tighter timeout and a fallback mechanism, the overall request latency was reduced by 150ms for the majority of users.

Managing Performance Overhead

Tracing every single request in a high-traffic environment can introduce CPU and memory overhead. To mitigate this, use sampling.

You can adjust the distributed_tracing.sampling_rate to capture only a percentage of traces. For example, setting this to 0.1 captures 10% of requests, which is usually sufficient to identify patterns without degrading performance.

To verify your sampling rate is working, run this NRQL query in the New Relic Query Console:

SELECT count(*) FROM Trace WHERE service.name = 'your-service-name' SINCE 1 hour ago

Compare the result against your total request volume from the Transaction event type to ensure the ratio matches your configuration.

Verification and Validation

1. Deploy your instrumented services to a staging namespace.
2. Trigger a request using curl or a load-testing tool.
3. Wait 30–60 seconds and search for the transaction in the New Relic UI.
4. Inspect the trace JSON to ensure the trace.id remains identical across all services in the chain.