Debugging Cross-Service Latency with Sentry Distributed Tracing

The Hidden Cost of Distributed Systems

When a user request bounces between five different services, where does the latency actually come from? Traditional logging shows each service's perspective, but it fragments the story. You end up with correlated logs that require manual stitching, or worse, guesswork during outages.

Sentry's Approach to End-to-End Visibility

Sentry's distributed tracing captures the complete journey of a request by propagating trace context through service boundaries. Each HTTP call, GraphQL operation, or background job becomes a span—a timed unit of work that carries metadata like operation name, tags, and error information. These spans nest into traces, creating a tree that mirrors your service architecture.

Unlike browser tracing which uses the JavaScript Performance API for frontend metrics, backend tracing requires SDK configuration to inject trace headers (like sentry-trace and baggage) into outgoing requests. Services receiving these headers extract the context and create child spans, building the trace automatically.

Configuring Trace Sampling for Production

The challenge isn't just capturing traces—it's managing the volume. Sentry uses a two-tier sampling approach: a trace sampling rate determines which requests get traced, and a span sampling rate controls how much detail each trace contains.

Default configuration might send 1% of requests as full traces, but you can adjust this per environment:

// Node.js SDK example
Sentry.init({
  dsn: 'https://[contact removed]/0',
  tracesSampleRate: 0.1, // 10% of requests
  // Or use a sampling function for dynamic control
  tracesSampleRate: (context) => {
    // Always trace errors
    if (context.transactionContext?.name?.includes('payment')) return 1.0;
    return 0.05;
  }
});

Working Example: Tracing a Payment Pipeline

Consider an e-commerce flow: api-gateway → auth-service → inventory-service → payment-service → notification-service. Each service needs the SDK to propagate trace context:

1. Service A (Incoming Request): Extract sentry-trace header, create transaction span
2. Service B (Outgoing Call): SDK automatically injects trace headers into HTTP client
3. Service B (Incoming Request): SDK extracts headers, continues trace under parent span

After deployment, verify traces appear in Sentry's Performance tab. Look for spans with unusually high data.http.response_time or spans that show database queries taking longer than expected. The flame chart visualization reveals bottlenecks without manual log correlation.

Trade-offs and Limitations

While powerful, distributed tracing has constraints. High-cardinality data—unique trace IDs for every request—can generate significant storage costs, especially at scale. The tracesSampleRate becomes critical: too low and you miss issues, too high and you pay for data you rarely need.

Some protocols require manual instrumentation. gRPC, message queues, or custom TCP connections don't automatically propagate HTTP-style headers. You'll need to manually extract and propagate trace context through these channels, adding complexity that pure HTTP tracing avoids.

Actionable Next Steps

Start with a single critical service boundary—perhaps your API gateway to your most complex business service. Configure a modest tracesSampleRate (2-5%), deploy, and verify traces appear in Sentry's Traces browser under Performance monitoring. Check that span data matches expected service boundaries and that errors within traces correlate properly. Once the pipeline works, incrementally add more services, always balancing sampling rates against your observability budget.