Add Automatic HTTP Client Tracing to Java Apps with the OpenTelemetry Java Agent

Problem and Takeaway

You need distributed traces for outbound HTTP calls from a Java service, but you cannot or do not want to modify application code. The OpenTelemetry Java Agent instruments standard HTTP clients (java.net.HttpURLConnection, Apache HttpClient 4/5, OkHttp 3/4, and others) at JVM startup, producing spans automatically. This guide shows how to attach the agent, configure export to a backend, verify traces appear, and revert cleanly if something breaks.

Prerequisites

• Java 8+ runtime (the agent supports Java 8 through 21; verify your version with java -version).
• An existing Java application launched via java -jar, java -cp, or a startup script you can edit.
• Network access from the application host to your trace backend (Jaeger, Zipkin, OTLP/gRPC, OTLP/HTTP, etc.).
• Write permission to the directory where you will store the agent JAR (typically /opt/otel or a project-local lib/ folder).

Download the Agent

Fetch the latest stable agent JAR from the OpenTelemetry releases page. As of this writing, version 2.14.0 is current; adjust the version in the URL if a newer release exists.

# Run on the application host
mkdir -p /opt/otel
cd /opt/otel
curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.14.0/opentelemetry-javaagent.jar
sha256sum opentelemetry-javaagent.jar  # verify against the checksum published on the release page

Risk: Downloading an unverified JAR can introduce supply-chain risk. Always compare the checksum.

Attach the Agent to the JVM

Modify the JVM command line to include -javaagent:/opt/otel/opentelemetry-javaagent.jar before the -jar or -cp argument. Example for a Spring Boot fat JAR:

java -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=order-service \
  -Dotel.traces.exporter=otlp \
  -Dotel.exporter.otlp.endpoint=http://collector:4318/v1/traces \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -jar /app/order-service.jar

If you use a startup script (systemd unit, Docker ENTRYPOINT, Kubernetes container command), insert the -javaagent flag there. The agent must appear before the main class or -jar; otherwise the JVM will not load it.

Configure Trace Export

The agent reads configuration from environment variables, system properties (-D flags), or a otel.properties file on the classpath. Environment variables are preferred for containerized deployments. Key variables:

For quick local testing without a collector, set OTEL_TRACES_EXPORTER=console to print spans to stdout.

Validate That HTTP Client Spans Appear

1. Start the application with the agent attached.
2. Trigger an outbound HTTP call (e.g., call a REST endpoint that itself calls an external API).
3. Check the collector logs or UI for a span named like HTTP GET or HTTP POST with attributes http.method, http.url, http.status_code, and net.peer.name.
4. If using the console exporter, look for JSON lines in the application stdout containing "name": "HTTP GET".

Diagnostic command (run on the application host while the app is running):

# Verify the agent loaded
jcmd $(pgrep -f order-service.jar) VM.system_properties | grep -i opentelemetry
# Expected output includes:
# otel.javaagent.enabled=true
# otel.service.name=order-service

If the jcmd output shows no OpenTelemetry properties, the -javaagent flag was either missing or placed after -jar.

Common Failure Modes and Recovery

Agent JAR Not Found or Unreadable

Symptom: JVM exits with Error opening zip file or JAR manifest missing.

Fix: Ensure the path in -javaagent is absolute and the file is readable by the user running the JVM (chmod 644 /opt/otel/opentelemetry-javaagent.jar).

Duplicate Spans from Programmatic SDK

Symptom: Two spans for each outbound call, one with instrumentation.name=opentelemetry-javaagent and another from your manual OpenTelemetrySdk initialization.

Fix: Remove the programmatic SDK initialization or disable agent instrumentation for the conflicting library via OTEL_INSTRUMENTATION_HTTPCLIENT_ENABLED=false (replace HTTPCLIENT with the relevant instrumentation module name). Only one instrumentation path should be active.

Exporter Connection Refused

Symptom: Application logs show io.grpc.StatusRuntimeException: UNAVAILABLE or HTTP 404/502 from the collector endpoint.

Fix: Verify OTEL_EXPORTER_OTLP_ENDPOINT points to a reachable collector. Test with curl -v http://collector:4318/v1/traces from the application host. Adjust firewall, service mesh, or DNS as needed.

Excessive Cardinality or Overhead

Symptom: High CPU/memory, or spans with high-cardinality attributes (full request bodies, dynamic URLs with IDs).

Fix: Set OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_HEADERS=false (default) and OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_REQUEST_BODY=false, OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_RESPONSE_BODY=false. Use OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT (default 128) to cap attributes per span.

Rollback Procedure

If the agent causes instability, remove the -javaagent flag from the startup command and restart the JVM. No code changes or classpath modifications are required because the agent is entirely external. The application will run exactly as it did before instrumentation was added.

Version Assumptions

• OpenTelemetry Java Agent 2.14.0 (released 2024-Q4).
• Java 17 or 21 LTS; the agent also supports Java 8, 11.
• OTLP HTTP/protobuf exporter to a collector running OpenTelemetry Collector 0.110+.

Behavior may differ on older agent versions (pre-2.0) which used different environment variable names (OTEL_EXPORTER_JAEGER_ENDPOINT instead of the unified OTLP variables).

Verification Checklist

• jcmd shows otel.javaagent.enabled=true.
• Collector receives spans with instrumentation.name=opentelemetry-javaagent.
• HTTP client spans include http.method, http.url, http.status_code.
• No duplicate spans for the same outbound call.
• Application latency and error rate unchanged after deployment.