Diagnosing Missing Spans in OpenTelemetry: A Practical Guide for Java, Go, Python, and .NET

Recognizable Condition

Your services generate traces — visible in logs or a console exporter — but the configured backend (Jaeger, Tempo, Zipkin, Datadog) shows zero spans or a fraction of the expected volume. Request throughput is normal, yet the trace UI remains empty or sparsely populated.

Cause and Diagnostic Quick Reference

Symptom	Likely Cause	Primary Check
No spans at all in backend	Exporter misconfigured (endpoint, auth, TLS)	Collector receiver logs; SDK export errors
Intermittent missing spans	Sampler dropping spans (ratio too low or parent-based decisions)	Inspect trace-flags on generated spans; verify sampler config
Spans missing from one service only	Context propagation failure (missing/incompatible propagators)	Verify W3C TraceContext and Baggage headers on ingress/egress
Spans reach collector but not backend	Batch processor timeout, backpressure, or queue saturation	Collector metrics: otelcol_exporter_send_failed_spans, batch_timeout events
Only root spans missing	Parent context not extracted on ingress (HTTP/gRPC server instrumentation gap)	Confirm propagator registration on server side; check auto-instrumentation attachment

Ordered Diagnostic Checks

1. Confirm SDK span generation — Enable the console exporter temporarily:

   OTEL_TRACES_EXPORTER=console

   Run a request and verify spans print to stdout. If none appear, the instrumentation itself isn’t initializing.
2. Validate exporter endpoint and authentication — Check environment variables:

   OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://collector:4318/v1/traces
   OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer <token>"

   Test reachability from the service host:

   curl -v http://collector:4318/v1/traces

   Expect HTTP 200/202. For gRPC, use grpcurl -plaintext collector:4317.
3. Rule out sampling as the culprit — Set a deterministic sampler for testing:

   OTEL_TRACES_SAMPLER=always_on

   Or, if using parent-based with a ratio, raise it:

   OTEL_TRACES_SAMPLER=parentbased_traceidratio
   OTEL_TRACES_SAMPLER_ARG=1.0

   Re-run the test. If spans now appear, adjust the production ratio (e.g., 0.1) rather than leaving always_on.
4. Verify global propagator registration — The SDK must have the same propagators on client and server. Set via environment (order doesn’t matter):

   OTEL_PROPAGATORS=tracecontext,baggage,b3

   In code (Java example):

   GlobalOpenTelemetry.setPropagators(ContextPropagators.create(
     TextMapPropagator.composite(
       W3CTraceContextPropagator.getInstance(),
       W3CBaggagePropagator.getInstance(),
       B3Propagator.injectingMultiHeaders())));

   Missing tracecontext breaks W3C compliance; missing b3 breaks legacy Zipkin integrations.
5. Inspect collector pipeline for send errors — Enable debug logging on the collector:

   --log-level=debug

   Search logs for Exporting spans and Span export succeeded/failed. Check the zpages extension (default :55679):

   curl http://collector:55679/debug/pipelinez
   curl http://collector:55679/debug/tracez

   Look for exporter_send_failed_spans counters and batch processor queue depth.
6. Check backend ingest health — Query the backend API directly, bypassing the UI:
   • Jaeger: GET /api/traces?service=my-svc&limit=1
   • Tempo: GET /api/search?q={service.name=\"my-svc\"}
   • Datadog: GET /api/v1/traces?query=service:my-svc
   If the API returns data but the UI doesn’t, the issue is UI/query-layer, not ingestion.
7. Correlate missing spans with specific libraries — If only HTTP client spans are missing but server spans appear, the client instrumentation may not be loaded. Verify auto-instrumentation agent attachment (Java -javaagent, .NET OpenTelemetry.AutoInstrumentation, Python opentelemetry-instrument) or manual @WithSpan / OpenTelemetry.instrumentation usage.

Fixes Tied to Findings

Exporter Endpoint / Authentication

Update the OTLP endpoint and headers at the SDK or collector level. For the SDK, set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_HEADERS. For the collector, ensure the otlp receiver listens on the expected port and path:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

Restart the collector after config changes. Validate with otelcol --config=config.yaml --dry-run.

Sampling Configuration

For debugging, OTEL_TRACES_SAMPLER=always_on guarantees every trace is exported. In production, use parentbased_traceidratio with a ratio like 0.1 (10%). Avoid traceidratio alone — it breaks parent-child consistency. Note: SDK versions before Java 1.20, Go 1.10, Python 1.15 default to parentbased_traceidratio with ratio 1.0; newer versions may default to always_on. Verify your version’s default.

Context Propagation

Ensure both client and server register the same propagators. The environment variable OTEL_PROPAGATORS=tracecontext,baggage,b3 works across languages. For manual setup, compose propagators explicitly (see Java snippet above). If using auto-instrumentation agents, they usually register tracecontext and baggage by default; add b3 if legacy systems send X-B3-* headers.

Collector Batch Processor Tuning

Defaults (timeout: 5s, send_batch_max_size: 512) are too conservative for high-throughput services. Increase in the collector config:

processors:
  batch:
    timeout: 30s
    send_batch_max_size: 2048
    send_batch_size: 1024

Add a memory_limiter processor before batch to prevent OOM under burst:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

Order matters: memory_limiter → batch → queued_retry (if used) → exporters.

Backend Backpressure

If collector metrics show exporter_queue_capacity saturated, scale backend ingesters (Jaeger collectors, Tempo distributors, Datadog agents). Enable the queued_retry processor in the collector to absorb temporary spikes:

processors:
  queued_retry:
    max_retries: 5
    retry_interval: 5s

Monitor otelcol_exporter_enqueue_failed_spans — a non-zero value means the queue is full and spans are dropped.

Missing Instrumentation

For manual instrumentation, ensure spans are created and ended correctly. In Go:

ctx, span := tracer.Start(ctx, "operation", trace.WithAttributes(attribute.String("key", "value")))
defer span.End()
// ... work ...

For auto-instrumentation, confirm the agent is attached. Java: -javaagent:path/to/opentelemetry-javaagent.jar. .NET: dotnet OpenTelemetry.AutoInstrumentation.dll or enable via OTEL_DOTNET_AUTO_ENABLED=true. Python: run with opentelemetry-instrument python app.py. If you mix manual SDK init with an agent, disable the agent (OTEL_JAVAAGENT_ENABLED=false) to avoid double-initialization conflicts.

Escalation Criteria

1. Spans generated locally but collector receives zero after 5 minutes — Network/firewall blocking, collector receiver misconfigured (wrong port/protocol), or SDK exporter pointing to wrong host. Verify with tcpdump or netcat on the collector port.
2. Collector receives spans but backend shows zero after 10 minutes — Backend ingest pipeline failure (schema mismatch, authentication, capacity). Check backend logs for 4xx/5xx responses from the collector exporter.
3. Spans present but missing attributes or service.name — Resource detector not running or OTEL_SERVICE_NAME unset. Set explicitly: OTEL_SERVICE_NAME=my-svc or in code Resource.create(Attributes.of(SERVICE_NAME, "my-svc")). Missing service.name breaks backend grouping.
4. Intermittent loss correlates with GC pauses or CPU saturation — Increase SDK buffer sizes (OTEL_BSP_MAX_QUEUE_SIZE, OTEL_BSP_MAX_EXPORT_BATCH_SIZE) and collector memory_limiter. Consider moving to a sidecar collector to isolate resource pressure.
5. All verified and spans still missing — Engage backend vendor support with collector debug logs (--log-level=debug), SDK debug logs (OTEL_LOG_LEVEL=debug), and a minimal reproducible test trace (see verification step below).

Verification Checklist

After each fix, run a quick end-to-end test:

otel-cli span --service test --name test-op --endpoint http://collector:4318/v1/traces

The span should appear in the backend within 30 seconds. Query the backend API directly (step 6 above) to confirm. If using tail-based sampling (tail_sampling processor), ensure all spans of a trace hit the same collector instance — configure your load balancer for consistent hashing on trace ID.

Limitations and Version Notes

• SDK defaults for propagators and samplers changed across versions (Java <1.20, Go <1.10, Python <1.15). Always check the specific version’s documentation.
• Auto-instrumentation agents may conflict with manual SDK initialization. Disable the agent when using programmatic setup.
• This guide covers OTLP/HTTP and OTLP/gRPC. Legacy exporters (Jaeger Thrift, Zipkin) have different configuration surfaces.
• Tail-based sampling requires trace-ID affinity; if your collector fleet is behind a round-robin LB, spans for the same trace may land on different instances, breaking tail sampling.

Note: This diagnostic guide reflects established OpenTelemetry behavior as of the 2024–2025 releases. Specific defaults, environment variable names, and metric identifiers may vary by SDK/collector version. Validate against your deployed versions before applying changes in production.