Architecture Note: Auto-Instrumenting Spring Boot Services with the Datadog Java APM Agent

You want distributed traces and service-level metrics from a Spring Boot fleet without touching application code, and you want to know exactly what the Datadog Java agent changes, what it can see, and what breaks when it goes wrong. This note covers the smallest design that meets that goal, the trust boundaries it introduces, and the conditions under which you should move to manual instrumentation instead.

Requirements

The design assumes three concrete needs:

• Per-request traces with spans for HTTP entry points, database calls, and outbound HTTP/gRPC/messaging calls, with no code changes.
• Trace context propagated between services so a request crossing two Spring Boot apps renders as one trace.
• Overhead low enough for production — a commonly cited planning figure is under ~5% CPU/latency overhead at modest sampling rates, but treat this as a budget to verify against your own load test, not a guarantee.

The smallest suitable design

Run the Datadog Java agent (dd-java-agent.jar) as a JVM -javaagent alongside each service, reporting to a local Datadog Agent (DaemonSet in Kubernetes, or a host agent on VMs). The agent performs bytecode instrumentation at class-load time for supported frameworks — Spring Web, JDBC, JMS/Kafka/RabbitMQ clients, gRPC, and common HTTP clients — so spans appear without annotations.

In Kubernetes, the smallest footprint is pod annotations plus the Admission Controller, which injects the agent and sets environment variables automatically:

# Pod template annotations (Admission Controller must be enabled cluster-side)
admission.datadoghq.com/java-lib.version: "v1"   # pin a reviewed version, not floating latest

# Environment variables the injector sets (shown for clarity)
DD_SERVICE=checkout-api
DD_ENV=prod
DD_VERSION=1.4.2
DD_TRACE_SAMPLE_RATE=0.1

Unified service tagging (DD_SERVICE/DD_ENV/DD_VERSION) is the one configuration decision worth getting right up front: it is what links traces, metrics, and logs, and renaming services later fragments historical data.

Trust and data boundaries

The agent sees everything your application sees. That is the point, and the risk.

• PII in spans. Default instrumentation can capture HTTP query strings, URL path parameters, and some request headers as span tags. If your URLs carry emails, tokens, or IDs subject to privacy rules, configure obfuscation/scrubbing rules (agent-level trace obfuscation or Datadog's sensitive-data scanning) before enabling this in production. Verify by inspecting raw spans for a few real requests.
• Agent-to-cluster traffic. Trace payloads leave the pod to the local Agent over localhost/UDS, then to Datadog intake over TLS. If your compliance posture requires data to stay in-region, confirm the correct Datadog site parameter is set.
• Agent conflicts. Only one -javaagent doing bytecode weaving should be active. Running the Datadog agent alongside another APM or a security agent (e.g., another vendor's instrumentation) can produce duplicate spans, class-loading errors, or startup failures. This is a version-compatibility item to verify per release — flag it for review before each agent upgrade.

Operational checks

After rollout, confirm instrumentation is actually working rather than silently absent:

1. Send a known test request that crosses at least two services. In the Datadog UI, open APM > Traces and confirm a single trace contains spans from both services (propagation working) and includes JDBC/HTTP client spans (auto-instrumentation working).
2. Check APM > Services for the service entry and confirm request/error/latency metrics are populating. If the service appears but spans are thin, the sampling rate may be dropping most traces.
3. On the host or in the pod, check the agent startup log line reporting which instrumentations were applied; a framework listed as "not applied" usually means an unsupported version.
4. Compare p99 latency and CPU on a canary pod against a control pod under the same load before fleet-wide rollout.

Failure modes

• Broken propagation: traces fragment into one-span-per-service. Common cause is a custom HTTP client or an async boundary (e.g., a hand-rolled thread pool) that the agent doesn't instrument. Fix: use a supported client, or add manual context propagation at that boundary.
• Startup failure after agent upgrade: bytecode weaving conflict. Mitigation: pin the agent version, roll out via canary, and keep the previous image tag available so a redeploy restores the old agent.
• Overhead spike: usually trace volume, not weaving. Lower DD_TRACE_SAMPLE_RATE and disable any instrumentation you don't need.

When this design stops fitting

Auto-instrumentation covers frameworks, not business logic. Move to manual spans (OpenTelemetry API or @Trace annotations) when you need spans around domain operations, custom attributes for business KPIs, or propagation across an unsupported messaging system. Also reconsider the javaagent approach entirely if your platform forbids JVM agents or if you need vendor-neutral instrumentation — in that case, the OpenTelemetry Java agent with a Datadog exporter is the alternative to evaluate.

Version note: agent behavior and configuration keys described here reflect the dd-trace-java 1.x line; confirm the exact flag names and supported framework versions against the release notes for the version you pin.