Teleport audit event forwarding: avoiding duplicate writes on retries

Direct Answer

Teleport provides a stable, unique event identifier in the uid field (a UUID) that persists across retries. There is no configuration option to switch from at-least-once to exactly-once delivery; idempotency must be managed by the downstream consumer or the external sink using this uid.

Likely Explanation and Behavior

Teleport's audit event forwarding is designed for reliability over strict uniqueness. While the event-handler plugin uses a cursor/checkpoint to track progress and minimize duplicates after service restarts, transient network failures or sink timeouts trigger automatic retries. Because these retries occur at the transport level, the same event payload—including its original uid—is sent multiple times.

The Stable Identifier

• The uid field: This is the primary key for deduplication. It is assigned at the moment of event creation.
• Avoid id or time: The id field typically refers to the event type (e.g., user.login), and timestamps may lack the precision required to distinguish between distinct events occurring in the same millisecond.

Implementation Patterns for Deduplication

Depending on your sink, use one of the following patterns to ensure a single write per event:

1. Idempotent Database Writes (Recommended)

• SQL Databases: Create a unique constraint on the uid column (and cluster name if using multiple clusters) and use INSERT ... ON CONFLICT DO NOTHING.
• NoSQL (DynamoDB): Use a conditional put operation with attribute_not_exists(uid).
• Search Engines (Elasticsearch/OpenSearch): Set the document _id to the Teleport uid. Subsequent deliveries will overwrite the existing document rather than creating a duplicate.

2. Distributed Cache Layer

Implement a "look-aside" cache (e.g., Redis) using SET NX with the uid as the key. The Time-to-Live (TTL) must exceed the maximum retry/backoff window of the Teleport event-handler to prevent duplicates from appearing after the cache entry expires.

Verification and Safety

To verify your implementation, use these scoped steps:

1. Capture a raw JSON audit event to identify the uid value.
2. Temporarily block connectivity to your sink to force a Teleport retry.
3. Verify that the redelivered event contains the identical uid.
4. Confirm that your sink logic ignores the second delivery of that specific uid.

Assumptions and Uncertainties

• Version Variance: Field names may vary slightly between Teleport versions or editions; verify the uid field exists in your specific release's audit schema.
• Multi-Cluster: If forwarding from multiple clusters, the uid may not be globally unique across clusters. Namespace your keys as cluster_name:uid.
• Session URIs: Note that audit_sessions_uri (session recording forwarding) operates on a different path than standard audit events.