Diagnosing Missing or Malformed HTTP Client Spans in OpenTelemetry Java Auto‑Instrumentation

What’s the Problem?

When you run a Java service with the OpenTelemetry Java agent, you expect every outbound HTTP call to appear as a client.http span in your trace viewer. In production you may notice:

• No HTTP client spans at all.
• Spans that lack http.method or http.url attributes.
• Spans that are truncated or missing child spans.
• Spans that are created but never reach the backend.
• High‑volume traffic where many spans are silently dropped.

These symptoms usually point to one of three layers: the agent configuration, the exporter/collector pipeline, or network/credential issues.

Recognizable Symptoms & Causes

Symptom	Typical Cause
No HTTP client spans	Agent not activated or httpclient instrumentation disabled
Missing attributes (e.g., http.method)	Attribute filters or older instrumentation version
Truncated spans or missing children	Exporter batch size or timeout too small; collector buffer overflow
Spans never reach backend	Incorrect OTLP endpoint, TLS handshake failure, or missing auth token
Spans dropped under load	Collector or exporter buffer overflow; default 512‑span batch size exceeded

Ordered Diagnostic Checks

1. Verify Agent Activation
   • Run the application with OTEL_LOG_LEVEL=debug.

     java -Dotel.javaagent.enabled=true -Dotel.javaagent.path=/opt/otel-javaagent.jar -jar myapp.jar

   • Search the logs for Instrumentation library activated and confirm client.http is listed.
   • If missing, ensure OTEL_AUTO_INSTRUMENTATION_ENABLED=true and that the agent JAR is on the classpath.
2. Check Instrumentation Configuration
   • Inspect OTEL_INSTRUMENTATION_HTTPCLIENT_ENABLED (defaults to true).
   • Verify any attribute exclusion patterns, e.g., OTEL_INSTRUMENTATION_HTTPCLIENT_ATTRIBUTES_EXCLUDE=*.password, do not strip http.method or http.url.
   • If you upgraded the agent, confirm the instrumentation version matches the otel.instrumentation.httpclient.version property.
3. Validate Exporter Settings
   • Confirm OTEL_EXPORTER_OTLP_ENDPOINT points to a reachable collector. Test with curl -v https://$OTEL_EXPORTER_OTLP_ENDPOINT (for HTTPS) or grpcurl -plaintext $OTEL_EXPORTER_OTLP_ENDPOINT v1.TracesService/Export.
   • Check TLS handshake: ensure the collector’s certificate is trusted or set OTEL_EXPORTER_OTLP_INSECURE=true for local testing.
   • Review batch size: OTEL_EXPORTER_OTLP_BATCH_SIZE=512 is default. Increase temporarily to 1024 and watch collector metrics.
4. Examine Collector Health
   • Run a local OTLP collector with --log-level debug.

     otelcol --config otelcol.yaml --log-level debug

   • Check the collector’s metrics endpoint (/metrics) for otelcol_exporter_otlp_sent_spans_total and otelcol_exporter_otlp_failed_spans_total.
   • Look for Failed to export spans errors in the logs indicating network or auth problems.
5. Test a Minimal HTTP Call
   • Add a simple HttpClient request in your code or use curl http://localhost:8080/health if the service exposes a health endpoint.
   • Query the trace backend (e.g., Jaeger UI) for a span with http.method=GET and http.url=/health.
   • If the span appears but attributes are missing, revisit attribute filter settings.

Fixes Tied to Findings

• Agent not active: Add -Dotel.javaagent.enabled=true to the JVM options or set OTEL_AUTO_INSTRUMENTATION_ENABLED=true in the environment.
• Instrumentation disabled: Remove or set OTEL_INSTRUMENTATION_HTTPCLIENT_ENABLED=false to true.
• Attribute filters mis‑configured: Clear exclusion patterns or explicitly include http.method and http.url via OTEL_INSTRUMENTATION_HTTPCLIENT_ATTRIBUTES_INCLUDE=http.method,http.url.
• Exporter batch too small: Increase OTEL_EXPORTER_OTLP_BATCH_SIZE and monitor memory usage. Also adjust OTEL_EXPORTER_OTLP_TIMEOUT if network latency is high.
• OTEL_EXPORTER_OTLP_ENDPOINT wrong or unreachable: Correct the URL, ensure DNS resolution, and verify TLS certificates. If using a token, set OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <token>.
• Collector buffer overflow: Scale the collector (more workers or larger buffers) or enable OTEL_EXPORTER_OTLP_BATCH_TIMEOUT to allow larger batches.
• High‑volume drop: Enable OTEL_EXPORTER_OTLP_LOG_LEVEL=debug in the collector, then increase OTEL_EXPORTER_OTLP_BATCH_SIZE or add a dedicated queue service.

Escalation Criteria

If after completing the above checks the HTTP client spans remain missing or malformed:

• Open a ticket with your observability or DevOps team, providing the agent logs, collector logs, and the environment variables used.
• Include metrics snapshots: otelcol_exporter_otlp_sent_spans_total, otelcol_exporter_otlp_failed_spans_total, and JVM heap usage.
• Request a deeper network trace or a temporary increase in collector resources to capture the missing payload.

Limitations & Practical Checks

• Enabling the agent in production may double the number of spans generated; monitor for increased egress traffic.
• Increasing batch size or buffer limits can raise memory consumption; validate heap metrics during testing.
• Changing the OTLP endpoint or credentials without updating the service’s environment can cause silent failures; always test connectivity with grpcurl before deploying.
• Some cloud providers auto‑rotate TLS certificates; if you see handshake errors, verify the collector’s cert chain is up‑to‑date.

Concrete Example

Below is a minimal application.yml snippet that enables HTTP client instrumentation, sets a custom endpoint, and ensures key attributes are exported.

# application.yml
otel:
  instrumentation:
    httpclient:
      enabled: true
      attributes:
        include: http.method,http.url
  exporter:
    otlp:
      endpoint: https://otel-collector:4317
      headers:
        Authorization: Bearer <YOUR_TOKEN>
      batch-size: 1024
      timeout: 30s

Deploy this configuration and restart the service. Verify the span appears in the backend with the expected attributes. If not, follow the diagnostics above.