Instrumenting a Java Microservice with Datadog APM: From Setup to Verification

Desired Outcome

Enable end‑to‑end distributed tracing for a Java microservice so that every request is automatically captured, visualised, and correlated across the service mesh in the Datadog APM UI.

Prerequisites

• Java 8 or newer runtime (JVM 8+).
• Build tool (Maven or Gradle) that produces a runnable JAR.
• Datadog Agent 7.x running on the same host or reachable over the network.
• Valid Datadog API key (DD_API_KEY) for the Agent to authenticate.
• Network access from the host to app.datadoghq.com (or the region‑specific endpoint) on TCP port 8126.
• dd‑java‑agent JAR (latest stable release for the Agent version).

Procedure Overview

1. Install or verify the Datadog Agent.
2. Download the dd‑java‑agent JAR and add it to the application classpath.
3. Configure environment variables or system properties for the agent.
4. Start the application and verify agent startup.
5. Trigger a sample request and confirm the trace appears in the UI.

Step 1: Install or Verify the Datadog Agent

On a Linux host, you can install the Agent via the official script. Run as root or with sudo:

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=<DD_API_KEY> bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

Verify the Agent is running and listening on the trace port:

systemctl status datadog-agent
# or
netstat -tulnp | grep 8126

Ensure the Agent version (e.g., 7.51.0) matches the dd‑java‑agent release you plan to use.

Step 2: Add the dd‑Java‑Agent JAR to the Classpath

Download the JAR from Datadog’s releases page or via Maven Central:

curl -L https://github.com/DataDog/dd-trace-java/releases/download/v0.86.0/dd-java-agent-0.86.0.jar -o dd-java-agent.jar

Place dd-java-agent.jar in a known directory (e.g., /opt/dd-java-agent/) or bundle it with the application JAR.

Step 3: Configure Environment Variables

Datadog recommends using JAVA_TOOL_OPTIONS so the agent loads automatically. Export the following on the host that runs the JVM:

export JAVA_TOOL_OPTIONS="-javaagent:/opt/dd-java-agent/dd-java-agent.jar"
# Optional: set the host and port if the Agent is not on localhost:8126
export DD_AGENT_HOST=<agent-host>
export DD_TRACE_AGENT_PORT=8126
# Optional: service name and environment for clearer UI grouping
export DD_SERVICE=orders-service
export DD_ENV=production

Alternatively, you can pass them as JVM arguments:

java -javaagent:/opt/dd-java-agent/dd-java-agent.jar \
     -DD_SERVICE=orders-service \
     -DD_ENV=production \
     -cp target/orders-service.jar com.example.Main

Step 4: Verify Agent Startup

When the application starts, the Agent writes a log line indicating it is listening for traces:

[dd-trace-java] INFO  Agent listening for traces on port 8126

You can tail the Agent log (/var/log/datadog/agent.log) or use datadog-agent status to confirm the trace endpoint is active and no errors are reported.

Step 5: Test Tracing

Send a request to the service (e.g., via curl or a browser):

curl -i http://localhost:8080/orders/123

Open the Datadog APM UI, navigate to the orders-service dashboard, and look for a new trace within 30–60 seconds. The trace should show the root span, any HTTP client spans, and internal method spans if the application uses the tracer API.

Expected Checks

• The Agent log shows a “Listening for traces” message and no error entries.
• Datadog UI lists the configured service (DD_SERVICE) under the APM section.
• At least one trace appears after a sample request, and the trace duration matches the request latency.
• The trace includes the expected tags (e.g., env:production, service:orders-service).

Recovery Options

If no traces surface:

1. Check connectivity – ping the Agent host and use telnet <agent-host> 8126 to confirm the port is reachable.
2. Validate environment variables – run env | grep DD_ inside the process shell or inspect the JVM’s -cp output for the -javaagent flag.
3. Inspect Agent logs – look for “Failed to connect to trace agent” or “Could not load agent jar” errors.
4. Confirm API key – ensure the key is active and not expired in the Datadog UI.
5. After correcting any issue, restart the application. The Agent will re‑initialize automatically.

Limitations & Practical Checklist

• Agent and dd‑java‑agent must share the same major version (e.g., Agent 7.x with dd‑java‑agent 0.86.x). Mismatches cause silent failures.
• Network firewalls must allow outbound traffic to Datadog’s ingestion endpoint on port 443 and inbound traffic on 8126 if the Agent is remote.
• Java 8+ is required; older JVMs lack the instrumentation support.
• Large application JARs may need additional memory; the agent can increase heap usage.

Before deployment, run this checklist:

• Verify dd-java-agent.jar is in the classpath.
• Confirm JAVA_TOOL_OPTIONS contains the agent flag.
• Check Agent status with datadog-agent status.
• Make a test request and watch the UI for a trace.
• If traces appear, you can proceed to enable distributed tracing across downstream services.