Datadog Agent v7.30+ Log Forwarding: Idempotency Key Persistence

Log Deduplication via API v2

The Datadog Agent (version 7.30 and later) introduces the use_idempotency_key configuration within logs_config. When enabled, the Agent attaches an Idempotency-Key header to requests sent to the /api/v2/logs endpoint to prevent duplicate entries during transient network failures.

Persistence Constraints

While this mechanism handles retries during a continuous Agent process lifecycle, there is uncertainty regarding the persistence of these keys across service restarts. If the Agent generates a key for a batch but restarts before receiving a successful acknowledgment from the API, the behavior of the subsequent retry determines whether a duplicate write occurs.

Given the current implementation of the datadog.yaml configuration, it is unclear how the Agent manages the state of pending keys during a crash or planned reboot.

• Does the Agent persist generated idempotency keys to disk to ensure deduplication after a restart?
• If keys are volatile, what is the expected behavior for log batches that were in-flight during a process termination?