Datadog Agent v7.30+ Log Forwarding: Idempotency Key Persistence

Question

Log Deduplication via API v2
The Datadog Agent (version 7.30 and later) introduces the use_idempotency_key configuration within logs_config. When enabled, the Agent attaches an Idempotency-Key header to requests sent to the /api/v2/logs endpoint to prevent duplicate entries during transient network failures.

Persistence Constraints
While this mechanism handles retries during a continuous Agent process lifecycle, there is uncertainty regarding the persistence of these keys across service restarts. If the Agent generates a key for a batch but restarts before receiving a successful acknowledgment from the API, the behavior of the subsequent retry determines whether a duplicate write occurs.

Given the current implementation of the datadog.yaml configuration, it is unclear how the Agent manages the state of pending keys during a crash or planned reboot.

Does the Agent persist generated idempotency keys to disk to ensure deduplication after a restart?
  If keys are volatile, what is the expected behavior for log batches that were in-flight during a process termination?

Tasadduq Burney · Answer

Answer
The Agent does not persist generated Idempotency-Key values to disk; they exist only in memory for the duration of the process.
Confirmed facts

Idempotency-Key is added to /api/v2/logs requests to deduplicate retries within a single Agent run.
  The configuration logs_config.use_idempotency_key must be true for the header to be sent.
  Successful API acknowledgment marks a batch as processed; unacknowledged batches are not guaranteed to be retained after a restart.

Likely explanation
Because the key is generated at the moment the batch is prepared and kept only in the Agent’s memory, a crash or planned reboot discards it. If the batch had not yet been acknowledged, the Agent will resend it on restart with a new key, potentially creating a duplicate entry in Datadog.
Verification steps

Enable debug logging for the log forwarder in datadog.yaml (set log_level: debug).
  Send a unique test log line with a known content.
  Stop the Agent before Datadog returns a 200 acknowledgment for that batch.
  Restart the Agent and let it re‑establish the connection.
  Check the Agent debug logs for whether the same Idempotency-Key is reused or a new one is generated.
  Search Datadog for the test line; if it appears twice, a duplicate write occurred.

Missing diagnostic detail needed: did the test batch receive a successful Datadog API acknowledgment before the Agent stopped? Confirm this to determine whether duplicates are expected.

Datadog Agent v7.30+ Log Forwarding: Idempotency Key Persistence

Log Deduplication via API v2

Persistence Constraints

1 answer

Answer

Confirmed facts

Likely explanation

Verification steps

Your perspective belongs here.

0 question comments