Diagnosing Missing or Incomplete Datadog APM Traces: A Step‑by‑Step Guide

Recognizable condition

You see gaps in APM: some requests produce full traces, others produce none; a downstream service’s spans disappear from an otherwise complete trace; or traces appear in one environment but not another. The symptom is not a total outage — it’s intermittent or partial loss.

Cause‑to‑signal map

Ordered checks

1. Verify tracer initialization – Ensure DD_SERVICE, DD_ENV, DD_VERSION (or library‑specific config) are set. Missing tags rarely drop traces but fragment the service map.
2. Confirm Agent APM receiver is enabled – Run on the host running the Agent:
   

   sudo datadog-agent status

   Look for the APM block: Receiver: running and non‑zero Traces received / Spans received counters. Requires permission to execute the Agent binary (usually root or the datadog-agent user).
3. Test reachability of the trace port – From the application host or container:
   

   nc -zv  8126

   or curl -v telnet://:8126. A timeout or refusal points to network policy, missing hostPort in Kubernetes, or Agent listening on a Unix socket instead of TCP.
4. Read the APM section of Agent status – The counters Traces received and Spans received should increase as traffic flows. Zero counters after step 3 mean the tracer isn’t reaching the Agent.
5. Search Live Span Search – In the Datadog UI, open APM → Live Span Search and filter by your service name. If spans appear here, ingestion succeeded.
6. Review ingestion controls and retention filters – Navigate to APM → Ingestion Controls and Retention Filters. Identify any filter that would discard your test trace (e.g., service:my‑service with a low keep‑rate).
7. Verify propagation headers at the split point – Capture request/response headers (e.g., via tcpdump or application logs) at the service where the trace ID changes. Look for traceparent (W3C) or X-B3-TraceId (B3). Missing or altered headers indicate a proxy, API gateway, or service mesh stripping them.

Fixes tied to findings

Zero traces received

• Enable APM in the Agent: ensure apm_config.enabled: true in datadog.yaml (or the equivalent Helm value).
• Point the tracer at the correct host/port: set DD_AGENT_HOST and DD_TRACE_AGENT_PORT (default 8126) in the application environment.

Containers or Kubernetes

• Expose the trace port via hostPort: 8126 on the Agent pod, or configure a Unix‑socket receiver (apm_config.receiver_socket) if your Agent version supports it. Verify the socket path matches the tracer’s DD_TRACE_AGENT_URL (e.g., unix:///var/run/datadog/apm.socket).

Tracer‑side sampling drops

• Raise the sampler rate (DD_TRACE_SAMPLE_RATE=1.0 for 100 %) or switch to Datadog’s ingestion‑control sampling which decides at the backend. Be aware higher rates increase ingest volume and cost.

Retention drops

• Adjust the retention filter that matches your service (increase keep‑rate or add a rule to keep key spans). Alternatively, enable single‑span ingestion for critical operations so they bypass trace‑level retention.

Split traces (broken propagation)

• Upgrade all tracers to versions supporting W3C traceparent and B3 simultaneously.
• Configure proxies/gateways (Envoy, NGINX, AWS ALB) to forward traceparent, tracestate, X-B3-* headers unchanged.
• If a service mesh is in use, enable its distributed‑tracing propagation (e.g., Istio trace_context propagation).

Escalation criteria

Open a Datadog support case when:

• Agent reports traces received but the UI behavior contradicts your configured ingestion or retention settings.
• Traces split only when passing through a specific proxy or service mesh, and header inspection shows the proxy stripping propagation headers.
• Tracer debug logs (enable with DD_TRACE_DEBUG=true) show repeated send failures despite confirmed connectivity.

Collect before contacting support:

• datadog-agent status output (full).
• Tracer debug logs covering a failing request.
• A trace ID that exhibits the problem (from Live Span Search).

Limitations & practical verification

• Default ports (8126 TCP, Unix socket path) and config keys are stable across recent Agent 7.x releases, but always verify against the official Agent APM configuration docs for your exact version.
• W3C trace‑context and B3 support varies by tracer language and version; mixed‑version fleets can split traces at hops.
• Sampling defaults differ per tracer (e.g., Java defaults to 10 %, Go to 100 %). Do not assume 100 % end‑to‑end capture.
• Socket‑based intake exists but the configuration key (apm_config.receiver_socket vs apm_config.socket_path) changes across Agent versions; confirm before switching from TCP.
• Retention percentages, quotas, and pricing change by plan; do not quote them from memory.

Quick verification: Emit a test trace from the application runtime (example for Python):

python -c "import ddtrace; tracer = ddtrace.tracer; span = tracer.trace('test.operation'); span.finish()"

Then, within a minute, search APM → Live Span Search for service:. If the span appears, ingestion works; if it appears only in Live Span Search but not in APM → Traces, the issue is a retention filter. If it never appears, revisit steps 1‑4.