Designing a Datadog Logs Pipeline Processor for PII Redaction

Problem

Applications emit logs that may contain personally identifiable information (PII) such as email addresses, credit‑card numbers, or social‑security numbers. Storing or analyzing these logs without masking exposes sensitive data and can violate regulations.

Useful Takeaway

By adding a single Datadog Logs pipeline processor that uses built‑in pattern matchers or custom regex to redact PII before logs are written to storage, you ensure downstream consumers only see masked values while keeping the raw data inside Datadog’s trusted environment.

Requirements

• Ingest logs from any source via the Datadog Agent or HTTP API.
• Detect fields that contain PII (e.g., email, credit card, SSN).
• Replace the detected value with a mask or token (e.g., [REDACTED]).
• Make the masked logs available for search, dashboards, and downstream processing.
• Provide visibility into processor health and error rates.

Smallest Suitable Design

The minimal solution is a single pipeline processor placed in the Datadog Logs pipeline:

1. Matcher – Use a built‑in pattern (email, credit_card, ssn) or a custom regex that targets the PII field.
2. Action – Apply a redact action that substitutes the matched value with a configurable mask string.
3. Routing – Send the processed log to the same Logs index used for raw ingest; no separate index is required.

This design adds no extra components outside Datadog and keeps the data flow linear.

Trust and Data Boundaries

The processor executes inside Datadog’s managed logging pipeline, which operates within the trusted Datadog environment. Raw logs never leave this boundary before redaction occurs. Consequently:

• Only masked values are written to persistent storage.
• Downstream consumers (analysts, alerting rules, third‑party integrations) automatically see redacted data.
• The original raw log is not retained unless you explicitly enable a separate raw‑log archive.

Operational Checks

Monitor the processor’s health with the following metrics available in Datadog Metrics Explorer:

• datadog.logs.pipeline.processor.applied – increments each time the processor successfully matches and redacts a field.
• datadog.logs.pipeline.processor.errors – increments when the matcher fails (e.g., regex compilation error) or the redact action cannot be applied.

Set alerts on:

• A sudden drop in applied volume (possible mis‑configuration or missing logs).
• A spike in errors (indicates a broken matcher).

Additionally, periodically sample a small set of raw logs (via the Agent’s debug output) and compare them with the stored logs to verify that PII fields are consistently masked.

Failure Modes

• Over‑redaction – A overly broad regex matches non‑PII text, causing loss of useful information. This appears as an unexpected increase in applied on fields that should remain unchanged.
• Under‑redaction – The matcher misses a PII pattern, leaving sensitive data in storage. This is detected by finding raw PII in sampled stored logs.
• Processor errors – If the processor is configured with fail_on_error, any matcher error causes the log to be dropped, increasing errors and reducing ingest volume.
• Performance impact – Complex regex patterns can raise CPU usage in the pipeline, typically under 5% but potentially higher with pathological patterns.

Design Change Triggers

Consider revisiting the design when:

• A new log schema introduces additional PII fields that are not covered by existing matchers.
• Regulatory updates require a different masking strategy (e.g., tokenization instead of static masking).
• Observed latency or CPU usage in the pipeline exceeds acceptable thresholds, prompting a move of redaction to the Agent side (dd-agent processing) to reduce pipeline load.

Example Configuration

The following JSON snippet shows how to define a pipeline processor that redacts email addresses. Place this in the Pipelines section of the Datadog Logs configuration UI or via the API.

{
  "name": "pii-email-redaction",
  "filter": "@service:web-app",
  "processors": [
    {
      "type": "matcher",
      "name": "email-matcher",
      "matcher": "email",
      "field": "message"
    },
    {
      "type": "redact",
      "name": "email-redact",
      "field": "message",
      "mask_string": "[REDACTED-EMAIL]"
    }
  ]
}

Adjust the filter to target the appropriate service or source, and add additional matcher‑redact pairs for other PII types as needed.

Verification Steps

1. Create a test pipeline using the configuration above.
2. Send a sample log line containing an email address via the Datadog Agent (e.g., echo '2026-10-14T12:00:00Z user login [contact removed]' | ddagent send-log).
3. In Log Explorer, verify that the stored message shows [contact removed] replaced with [REDACTED-EMAIL].
4. Check the datadog.logs.pipeline.processor.applied metric; it should increase by one for the test log.
5. Confirm that datadog.logs.pipeline.processor.errors remains near zero.
6. Optionally, set the processor to fail_on_error, inject a log that triggers a regex error (e.g., malformed custom pattern), and observe that the log is dropped and the error metric increments.

Limitations and Practical Checks

• Redaction is irreversible; ensure your retention and audit policies are satisfied with masked data.
• If the Agent is configured to drop logs before they reach Datadog (e.g., via exclude_at_match), the pipeline processor will never see those logs, creating a coverage gap. Verify Agent forwarding settings.
• Test complex regex patterns on a representative log sample in a non‑production pipeline before enabling them in high‑traffic environments to avoid unexpected CPU spikes.