Linking Datadog Logs to APM Traces with Trace ID Injection

Problem: Logs alone don’t show the full request flow

When a service returns an error, you often see the exception message in Datadog Logs but lack the context of the upstream calls that led to it. Switching manually between Logs and APM to find the related trace is time‑consuming and error‑prone, especially during high‑traffic incidents.

Thesis: Inject the Datadog trace ID into every log line to enable one‑click navigation

Datadog automatically correlates logs and traces when a log entry contains the trace.id field (or the legacy dd.trace_id field). By ensuring your application logs include this field, the Logs UI shows a “View trace” button that jumps directly to the corresponding APM trace.

How to inject the trace ID

Datadog’s tracing libraries (e.g., ddtrace for Python, datadog-trace-rb for Ruby) make the current trace ID available through environment variables or API calls. You can add it to your logging format with minimal code changes.

Automatic injection (recommended)

If you configure the Datadog Agent to collect logs and enable logs_injection: true in datadog.yaml, the Agent will automatically enrich log lines with trace.id and span.id when the application writes to stdout/stderr and the Agent is tailing those streams. This works for any language that writes logs to the console.

# /etc/datadog-agent/datadog.yaml (requires root or dd-agent user)
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-app
    source: python
    log_injection: true

When log_injection is enabled, the Agent adds the fields at ingestion time; you do not need to modify application code.

Manual injection (when you cannot use the Agent)

If you ship logs via a custom pipeline (e.g., Fluent Bit, Vector), you can add the trace ID yourself. Most Datadog tracing libraries expose the current trace ID via an environment variable (DD_TRACE_ID) or a runtime API.

Example for a Python Flask app using ddtrace and the standard logging module:

import os
import logging
from ddtrace import tracer

class TraceIDFilter(logging.Filter):
    def filter(self, record):
        # tracer.current_trace_id() returns 0 when no trace is active
        record.trace_id = tracer.current_trace_id() or 0
        return True

logger = logging.getLogger('my-app')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.addFilter(TraceIDFilter())
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s trace_id=%(trace_id)d')
handler.setFormatter(formatter)
logger.addHandler(handler)

# Example route
from flask import Flask
app = Flask(__name__)

@app.route('/')
def home():
    logger.info('Request received')
    return 'OK'

The resulting log line looks like:

2026-09-26T15:10:00Z INFO Request received trace_id=1234567890abcdef

Worked example: Verifying the link in the UI

1. Generate a test log: Run a request to your service (e.g., curl http://my-app.local/) while ensuring tracing is active.
2. Check the Logs UI: In Datadog, navigate to Logs → Search and query:
   service:my-app @trace_id:1234567890abcdef (replace the trace ID with the one you saw in the log line).
3. Confirm trace linkage: The log entry should display a “View trace” button. Clicking it opens the APM trace view showing the full request timeline.
4. Optional CLI verification: Using the Datadog CLI (datadog logs query) with the same filter returns the log entry and shows the associated trace ID in the output.

If the button does not appear, verify that:

• The log contains a numeric trace_id field (not a string).
• The Agent version is ≥7.30 (required for automatic injection).
• Trace ingestion is enabled for the service (apm_config: enabled: true in datadog.yaml).

Trade‑offs and limitations

Injecting trace IDs improves debuggability but introduces considerations:

• Log size increase: Each line gains roughly 20‑30 bytes for the trace ID field. At high volume this can raise ingestion costs.
• Cardinality risk: If you index trace_id as a tag for filtering, its high cardinality (unique per request) can degrade query performance. Keep it as a searchable field only; do not add it to the index list unless you specifically need to filter on it.
• Agent resource usage: Automatic injection requires the Agent to parse each log line, adding minimal CPU overhead. Monitor Agent metrics (datadog.agent.process) after enabling the feature.

Practical check: compare daily log volume before and after enabling injection via the Usage → Logs page. If the increase exceeds your budget, consider sampling logs (sample_rate in the Agent config) or limiting injection to specific services.

Actionable closing

1. Enable log_injection: true in the Datadog Agent for services that write logs to stdout/stderr, or add a logging filter that inserts trace_id from your tracing library.
2. Deploy the change, then run a representative request and verify the “View trace” button appears in the Logs UI.
3. Monitor log ingestion volume and Agent CPU for any unexpected spikes; adjust sampling or retention policies if needed.
4. Create a saved log query (e.g., service:my-app @trace_id:*) and pin it to a dashboard for quick access during incident response.

By making the trace ID a first‑class part of your log output, you turn a static error message into a gateway to the full request context—reducing mean‑time‑to‑understand and helping your team resolve issues faster.