Bringing End‑to‑End Tracing into a Spring Boot Service with Datadog’s OpenTelemetry Exporter

Concrete Problem: Missing End‑to‑End Trace Visibility

When a microservice is built with Spring Boot, developers often rely on the built‑in spring-boot-starter-actuator for basic metrics. But as services grow, the need for distributed tracing becomes critical. Without a tracing solution, a request that traverses several services can look like a black‑box: you see the latency in the metrics, but you can’t follow the actual path the request took.

Thesis: The Datadog OpenTelemetry Exporter Gives Seamless, Low‑Overhead Tracing

Datadog ships a fully supported OpenTelemetry exporter that forwards spans straight to the Datadog Agent. By configuring a few environment variables or system properties you can get end‑to‑end visibility in the Datadog UI without touching the agent’s code or adding a separate tracing library to your service. The exporter batches spans and reuses a single HTTP connection, keeping the runtime overhead minimal even under high load.

How It Works

1. Instrument the Service – Either use OpenTelemetry’s auto‑instrumentation Java agent or add the SDK manually. The exporter is a plug‑in that can be enabled through a simple configuration change.

2. Export to the Agent – The exporter talks to the Datadog Agent via the trace API (http://agent-host:8126/v1/trace). The Agent receives the spans, enriches them with metadata, and forwards them to Datadog’s backend.

3. Visualize in Datadog – Once the spans reach Datadog, the UI automatically aggregates them, links them to metrics and logs, and provides a full trace view.

Concrete Spring Boot Example

Below is a minimal Spring Boot application that demonstrates the setup. The example uses manual SDK configuration to keep the code explicit, but the same environment variables work with the auto‑instrumentation agent.

@SpringBootApplication
public class TracingDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(TracingDemoApplication.class, args);
    }

    @Bean
    public OpenTelemetrySdk openTelemetrySdk() {
        // Create a tracer provider with the Datadog exporter
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(SimpleSpanProcessor.create(
                DatadogGrpcExporter.builder()
                    .setEndpoint("http://localhost:8126")
                    .build()))
            .build();

        return OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .buildAndRegisterGlobal();
    }
}

@RestController
class DemoController {
    private final Tracer tracer = OpenTelemetry.getGlobalTracer("demo");

    @GetMapping("/hello")
    public String hello() {
        Span span = tracer.spanBuilder("hello-span")
            .setSpanKind(SpanKind.SERVER)
            .startSpan();
        try (Scope scope = span.makeCurrent()) {
            // Simulate work
            Thread.sleep(50);
            return "Hello, world!";
        } catch (InterruptedException e) {
            span.recordException(e);
            Thread.currentThread().interrupt();
            return "Error";
        } finally {
            span.end();
        }
    }
}

**Environment Variables** – Add the following to your container or VM. They work with both manual SDK setup and the auto‑instrumentation agent.

• DD_AGENT_HOST=localhost – Hostname of the Datadog Agent.
• DD_AGENT_PORT=8126 – Agent trace port (default).
• DD_SERVICE=my-spring-service – Service name shown in Datadog.
• DD_ENV=production – Optional environment tag.

**Java System Properties** – If you prefer properties, set:

• -Ddd.service=my-spring-service
• -Ddd.env=production
• -Ddd.agent.host=localhost
• -Ddd.agent.port=8126

Verification Checklist

1. Run the Datadog Agent locally or in the same network. It must be listening on port 8126.
2. Start the Spring Boot app with the environment variables set.
3. Open http://localhost:8080/hello a few times.
4. Navigate to APM → Traces in the Datadog UI and confirm the root trace and the hello-span child appear.
5. Optional: In the Agent’s metrics tab, look for otel_exporter_datadog_sent_spans to see that spans are being transmitted.
6. To test the exporter path directly, POST a JSON span to http://localhost:8126/v1/trace and verify it shows up in the UI.

Trade‑Offs and Limitations

• Agent Dependency – The exporter relies on the Datadog Agent. If the Agent is down or misconfigured, spans will be silently dropped.
• Network Restrictions – The service must be able to reach localhost:8126 (or the configured host). In environments with strict egress policies, you may need to open the port or use a sidecar.
• Batch Size Tuning – The default batch size is 1000 spans or 5 seconds, whichever comes first. For very high‑throughput services, this may need adjustment to avoid latency spikes.
• CPU Overhead – The OpenTelemetry SDK adds a small CPU cost. Monitor otel_exporter_datadog_dropped_spans and JVM GC metrics after enabling tracing.
• No Automatic Context Propagation – If you use manual instrumentation, you must manage context propagation (e.g., using Scope) unless you use the auto‑instrumentation agent.

Actionable Next Steps

1. Deploy the Datadog Agent on the same host or in a dedicated sidecar container.
2. Add the environment variables or system properties to your deployment manifests.
3. Instrument your service with the snippet above or enable the Java agent.
4. Run a short load test (e.g., k6 run script.js) and monitor the sent_spans metric to confirm throughput.
5. Use Datadog’s Service Map to verify that your service appears correctly and that downstream calls are linked.
6. Iterate on batch size and timeout if you notice latency spikes or dropped spans.

By following this approach you get end‑to‑end trace visibility, tight integration with Datadog metrics and logs, and a future‑proof path that works across Java, Python, Go, and other languages that Datadog supports.