Preventing Trace Fragmentation: Implementing OTel Context Propagation

The Problem: Broken Traces in Distributed Systems

When a request travels through multiple microservices, it often appears in your observability backend as a series of disconnected spans rather than a single, continuous trace. This fragmentation happens because the Trace ID—the unique identifier that links all operations in a request—is not being passed from the calling service to the receiving service.

To fix this, you must implement Context Propagation. This is the process of injecting trace metadata into the transport layer (usually HTTP headers) at the source and extracting it at the destination to maintain the causal relationship between services.

How Context Propagation Works

OpenTelemetry uses a TextMapPropagator to handle the serialization of context. By default, OTel implements the W3C Trace Context standard. This standard uses two primary headers:

• traceparent: Contains the version, trace ID, parent span ID, and trace flags.
• tracestate: Carries vendor-specific contextual information.

When Service A calls Service B, the OTel SDK "injects" these headers into the outgoing request. Service B then "extracts" these headers to start its own span as a child of the incoming trace ID.

Practical Implementation: Java SDK

While the OpenTelemetry Java Agent provides zero-code instrumentation for common libraries (like Spring Boot or OkHttp), manual propagation is required when using custom transport layers or complex asynchronous patterns.

Manual Injection and Extraction Example

The following example demonstrates how to manually propagate context using the GlobalOpenTelemetry instance. This assumes you have opentelemetry-api and opentelemetry-sdk in your classpath.

// 1. Injecting context into an HTTP request (Service A) 
// Run this in the service initiating the call
TextMapSetter<HttpRequest> setter = (carrier, key, value) -> 
    carrier.setHeader(key, value);

GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
    .inject(Context.current(), httpRequest, setter);

// 2. Extracting context from an incoming request (Service B)
// Run this in the receiving service's middleware or interceptor
TextMapGetter<HttpRequest> getter = new TextMapGetter<HttpRequest>() {
    @Override
    public Iterable<String> enumerable(HttpRequest carrier) {
        return carrier.headers().keySet();
    }

    @Override
    public String get(HttpRequest carrier, String key) {
        return carrier.getHeader(key);
    }
};

Context extractedContext = GlobalOpenTelemetry.getPropagators()
    .getTextMapPropagator()
    .extract(Context.root(), httpRequest, getter);

// Wrap the execution in the extracted context to ensure child spans are linked
try (Scope scope = extractedContext.makeCurrent()) {
    Span span = GlobalOpenTelemetry.getTracer("my-service").spanBuilder("process-request").startSpan();
    try {
        // Business logic here
    } finally {
        span.end();
    }
}

Execution Requirements

• Permissions: The application must have network egress permissions to send traces to the OTel Collector.
• Placeholders: Replace HttpRequest with your specific HTTP client/server object (e.g., HttpServletRequest).
• Risk: Failing to call scope.close() (or using try-with-resources) will lead to Context Leaks, where subsequent unrelated requests are incorrectly tagged with the previous request's Trace ID.

Common Pitfalls and Limitations

The Async Gap (ThreadLocal Loss)

In Java, the OTel SDK stores the current span in a ThreadLocal variable. When you switch threads—such as using CompletableFuture, parallelStream(), or a custom ExecutorService—the context is lost. The new thread will start a brand new trace, breaking the chain.

Solution: You must wrap your executor. Instead of using a raw ExecutorService, use the OTel wrapper to propagate the context to the worker thread:

Executor wrappedExecutor = Context.taskWrappingExecutor(originalExecutor);
wrappedExecutor.execute(() -> { 
    // Trace context is now preserved here
});

Sampling Overhead

Capturing every single request in a high-traffic environment can degrade CPU performance and overwhelm your storage backend. Use a Sampling Strategy (e.g., ParentBased(root=TraceIdRatioBased(0.1))) to only record 10% of traces while ensuring that if a parent service decides to sample a request, all downstream services follow suit.

Verifying the Propagation

To confirm that propagation is working correctly, perform the following checks:

1. Header Inspection: Use a tool like Wireshark or a proxy to verify that the outgoing request from Service A contains the traceparent header.
2. Backend Visualization: Open your tracing backend (e.g., Jaeger). Search for a specific Trace ID. You should see a single Gantt chart showing the request entering Service A and then moving into Service B, rather than two separate traces.
3. Log Correlation: If you have integrated OTel with your logging framework, verify that the trace_id in the logs of Service A matches the trace_id in the logs of Service B for the same request.