Structure Datadog Logs at Ingestion and Link Them to Traces

Raw logs make incidents harder than they need to be

When an error spikes in production, the first place most teams look is the log stream. If those logs arrive as unstructured text, engineers spend minutes grepping for timestamps, service names, and error codes instead of diagnosing the failure. Worse, if the log entry does not carry a trace ID, there is no reliable way to jump from a log line to the distributed trace that shows the full request path.

The useful takeaway is simple: configure Datadog log ingestion to extract structured fields immediately, and make sure trace IDs are present so logs and traces stay connected. You do not need to rewrite application code to get there.

Extract fields with regex and JSON parsers

Datadog accepts logs over HTTP, TCP, and UDP, and applies pipeline processors at intake. For JSON logs, the JSON field extractor turns keys like `status_code` or `duration_ms` into searchable attributes. For free-form text, a regex extractor can capture patterns such as `request_id=([A-Za-z0-9-]+)`.

In the Datadog app, go to Logs > Pipelines, create a pipeline for your source, and add a processor. Choose "JSON log" if your application emits JSON, or "Regex" for delimited text. Map extracted values to fields such as `http.status_code` or `service.name`. These fields become available in the Logs UI and can be used to create metrics or monitors.

Worked example: a Python service log line

Suppose your service emits this JSON line:

{"timestamp":"2026-10-20T06:15:00Z","service":"checkout","status_code":500,"duration_ms":1240,"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","message":"payment gateway timeout"}

Add a JSON processor in the pipeline and map:

Source key	Datadog field	Use
`status_code`	`http.status_code`	Filter errors and build error-rate monitors
`duration_ms`	`http.server_duration`	Create latency dashboards
`trace_id`	`trace.id`	Link the log to its distributed trace

After saving, search `http.status_code:500 AND service:checkout` in the Logs UI. The extracted fields appear as filters, and the trace ID should be clickable if trace correlation is enabled.

Connect logs to distributed traces

Datadog trace correlation links log entries to traces when the log contains a trace ID that matches an active span. In APM, Datadog uses the `trace.id` and `span.id` conventions. If your application instruments with the Datadog tracing library, these IDs are usually injected automatically. If not, add them to your structured log output.

To verify correlation, open a trace in the APM service view, select a span, and look at the Logs tab. You should see log lines from the same service and time window that share the trace ID. Alternatively, search a trace ID in Logs and confirm the associated trace view appears.

Watch ingestion volume and retention

Structured extraction does not reduce the volume of bytes ingested. Broad index patterns and high-cardinality fields can increase cost and slow searches. Datadog lets you set index retention and sampling rules to control intake without turning off collection.

Before extracting every field, identify which attributes your team actually queries. Extract `status_code`, `trace_id`, and `service`, but avoid indexing random UUIDs or per-request tokens as fields.

Practical verification

1. In the Logs UI, search for a known structured field value, for example `http.status_code:500`.
2. Open a matching log and confirm the linked trace ID appears in the associated trace view.
3. Check the Ingestion dashboard under Infrastructure Logs for real-time throughput, error rates, and indexed byte volume.

If the trace view does not appear, confirm that the trace ID format matches what your APM agent expects and that the service is sending traces to the same Datadog organization.

Closing

Start with one pipeline, one source, and three fields. Verify that a log line leads to a trace, then expand extraction only where it reduces investigation time. Structured logs and trace correlation work best when they remove steps during an incident, not when they add more data to search.