Implementing Distributed Tracing in Datadog APM for Microservices: A Step‑by‑Step Guide

Problem: Lost visibility across service boundaries

When a request flows through several microservices, latency spikes or errors often appear only in one service’s logs. Without a way to stitch those logs together, engineers spend time correlating timestamps and guessing which downstream call caused the issue.

Takeaway

By enabling Datadog APM’s distributed tracing, each request carries a trace context (trace‑id, parent‑id, sampling‑priority) via HTTP headers. The Datadog Agent collects spans from every instrumented service and reassembles them into a unified trace, letting you see the full call chain in the Trace Search and Service Map views.

1. Prerequisites

• Datadog account with API key and site (e.g., datadoghq.com or us3.datadoghq.com).
• A running Datadog Agent (version 7.x or later) that can send APM data.
• Language‑specific Datadog tracing library for each service (Java dd‑trace‑agent, Python ddtrace, Go ddtrace‑opentelemetry, etc.).

Agent installation (Docker example)

docker run -d --name dd-agent \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -e DD_API_KEY=<DD_API_KEY> \
  -e DD_SITE=<DD_SITE> \
  -e DD_LOGS_ENABLED=true \
  -e DD_APM_ENABLED=true \
  -e DD_APM_NON_LOCAL_TRAFFIC=true \
  datadog/agent:latest

Replace <DD_API_KEY> and <DD_SITE> with your values. The agent must be able to reach the Datadog intake endpoints.
2. Instrument the services
Add the tracing library as a dependency and enable automatic instrumentation. Below is a minimal Spring Boot example; similar steps apply to other languages.
Maven dependency
<dependency>
    <groupId>com.datadoghq</groupId>
    <artifactId>dd-trace-ot</artifactId>
    <version>1.23.0</version>
</dependency>

Application properties
# Enable Datadog tracing
dd.trace.enabled=true
# Service name as it should appear in Datadog
dd.service=order-service
# Environment (prod, staging, dev)
dd.env=staging
# Version (optional but useful for filtering)
dd.version=1.4.0
# Propagate trace context via HTTP headers
dd.propagation.extract.style=datadog

When the service starts, the library injects the x-datadog-trace-id, x-datadog-parent-id, and x-datadog-sampling-priority headers into outgoing HTTP calls. Ensure no middleware or proxy strips these headers.
3. Verify the trace appears in Datadog

Trigger a cross‑service request (e.g., call GET /orders on the order service, which internally calls the inventory service).
In Datadog UI, go to APM → Trace Search.
Set the time window to the last few minutes and filter by service:order-service.
You should see a trace with two spans: one for the order service (root) and a child span for the inventory service. Click the trace to view the waterfall.
Check the Tags tab on each span; you can add custom tags (e.g., user.id:12345) via the API ddtrace.set_tag to enable high‑cardinality filtering.
Open APM → Service Map to see the directed edge from order‑service to inventory‑service, confirming the dependency was discovered automatically.

4. Trade‑offs and limitations

Sampling: By default, the agent samples traces at a fixed rate (e.g., 1 %). High‑volume services may drop many traces, making rare latency spikes invisible. You can adjust the sampling priority via DD_TRACE_SAMPLE_RATE or implement rule‑based sampling in the agent configuration (apm_config: sample_rate: 0.1). Verify the effective rate in APM → Trace Analytics → Sampling rate.
Overhead: Adding custom spans introduces a few microseconds of latency per span. Profile your application under load to ensure the overhead stays within your SLOs.
Header propagation: If a sidecar proxy, API gateway, or load balancer strips the x-datadog-* headers, the trace breaks. Test by adding a sidecar that logs incoming headers; ensure the three Datadog headers are present.

Actionable closing
1. Deploy the agent and tracing library to all services in the call chain.
2. Verify a test request produces a multi‑service trace in Trace Search.
3. Check the Service Map for the expected dependencies.
4. Review the sampling rate; increase it if you notice missing traces for critical paths.
5. Monitor the agent’s health (agent status or the Infrastructure tab) to confirm heartbeats and APM throughput.
Once these steps are confirmed, you’ll have continuous, query‑level visibility into how requests travel across your microservices, making it easier to pinpoint latency outliers and error sources.