Build a Datadog Log Pipeline for Custom Log Formats with Grok, Remappers, and Category Processor

Desired outcome

Logs from a custom application arrive in Datadog with structured attributes: the timestamp is mapped to the reserved @timestamp field, the log level becomes status, service and host are set correctly, and a severity facet is available for filtering. This enables dashboards, monitors, and log‑based metrics without needing to match on the raw message string.

Prerequisites

• A Datadog organization with Log Management enabled.
• A working collection path for the logs (Datadog Agent tailing files, container stdout, or HTTP intake).
• A user role that includes logs_pipeline_edit permission.
• Several representative sample lines, including edge cases such as missing fields or alternative timestamp formats.

Procedure

1. Create a new pipeline

1. In the Datadog app, navigate to Logs → Pipelines and click New Pipeline.
2. Give the pipeline a descriptive name, e.g., myapp-custom-log.
3. Set the Filter to match your logs. For example, if you tag logs with source:myapp or they come from a service named myapp, use source:myapp or service:myapp as the filter expression.
4. Click Create to open the pipeline editor.

2. Add a Grok parser

1. Click Add Processor → Grok Parser.
2. Paste a sample log line into the Sample log field. Example line:

   [2023-08-15T12:34:56Z] INFO service=api host=web01 message="Request processed"

3. In the Grok pattern box, enter a pattern that matches the line. Datadog’s Grok dialect supports common patterns; a working pattern for the example is:

   %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} service=%{WORD:service} host=%{WORD:host} message=%{GREEDYDATA:message}

4. Click Preview. The editor should show extracted attributes: timestamp, level, service, host, and message. If any field is missing, adjust the pattern (e.g., make the message part optional with %{GREEDYDATA:message?}).
5. Save the processor.

3. Map extracted attributes to reserved fields

1. Add an Attribute Remapper processor after the Grok parser.
2. Configure the following mappings:
   • timestamp → @timestamp (type: Date)
   • level → status (type: String)
   • service → service (type: String)
   • host → host (type: String)
3. Enable Overwrite if exists for each mapping to ensure the reserved fields are set even if a default value was previously applied.
4. Click Preview to verify that @timestamp shows the parsed date, status shows the log level (e.g., INFO), and service/host appear.
5. Save the remapper.

4. Add a Category processor for severity faceting

1. Add a Category Processor after the remappers.
2. Set the Source attribute to status.
3. Define categories that map log levels to Datadog severity levels. Example mapping:
   • ERR, ERROR → error
   • WARN, WARNING → warn
   • INFO → info
   • DEBUG → debug
4. Optionally, set a Default category of info for any unmatched values.
5. Click Preview to see a new attribute ddcategory (or whatever name you chose) with the appropriate severity value.
6. Save the processor.

5. Save the pipeline and verify ordering

After adding all processors, click Save Pipeline. The pipeline appears in the global list; ensure its position is above any less‑specific pipelines that might otherwise match the same filter. Only the first matching pipeline processes a log, so keep filters mutually exclusive.

Expected checks

In‑pipeline preview

Before saving, use the Sample log area to step through each processor and confirm the output matches expectations. This catches syntax errors in the Grok pattern or mis‑ordered remappers.

Live verification

1. Send a test line through your configured collection method. For example, if you use the HTTP intake, run:

   curl -X POST 'https://http-intake.logs.datadoghq.com/v1/input/' \
        -H 'Content-Type: application/json' \
        -d '{"ddsource":"myapp","ddservice":"myapp","hostname":"web01","message":"[2023-08-15T12:34:56Z] INFO service=api host=web01 message=\"Request processed\""}'
   

2. Open Log Explorer, apply a filter for source:myapp (or your chosen filter).
3. Confirm that the log entry shows facets for status, service, host, and the severity category you defined.
4. Check that the original raw message remains available under the message attribute.

Pipeline health

On the Pipelines page, view the Logs processed counter for your pipeline. A non‑zero count indicates logs are reaching the pipeline. Use Live Tail to see incoming logs in real time and verify they are tagged with the pipeline name.

Recovery and rollback options

Pipeline changes affect only logs ingested after saving; already‑indexed logs retain their original attributes. If a mistake is discovered:

• Disable the pipeline: toggle the switch off on the Pipelines page. Logs will bypass the pipeline and arrive with default attributes, allowing you to fix the configuration.
• Reorder pipelines: drag the pipeline to a different position if another pipeline was incorrectly intercepting logs.
• Edit processors: open the pipeline, adjust the Grok pattern, remapper mappings, or category definitions, then save.
• Version‑controlled rollback: if you manage pipelines as code (e.g., using Terraform’s datadog_log_pipeline resource), revert to a previous commit and run terraform apply. This restores the exact processor list and filter.

Limitations and practical tips

• Pipeline changes are not retroactive; historical logs must be re‑ingested if you need them parsed.
• Datadog’s Grok dialect may differ from other implementations; test patterns directly in the pipeline editor rather than copying from external sources.
• Overlapping filters cause only the first matching pipeline to run. Keep filters mutually exclusive or use nested pipelines under a common parent filter.
• Date remapping is order‑sensitive: ensure the @timestamp remapper runs before any processor that depends on the timestamp (e.g., a category processor that uses time‑based buckets).
• Review current UI labels, plan limits, and pricing in the Datadog documentation before publishing any claims about feature availability.