DynamoDB Streams and Lambda: Duplicate Record Side-Effects in Local Development

Recommended pattern: idempotency key before side effects

Derive an idempotency key from each DynamoDB Streams record, preferably the record's eventID. Before running any side effect, write the key to an idempotency table using a conditional put with attribute_not_exists(eventId). If the condition fails, the record was already processed; skip the side effect. If it succeeds, execute the side effect and commit the marker in the same transaction when possible.

Why marker placement matters

Writing the idempotency marker after the side effect leaves a retry window where a duplicate record can repeat the side effect. Lambda concurrency limits and reserved concurrency change retry timing but do not eliminate duplicate deliveries.

Confirmed behavior versus local uncertainty

DynamoDB Streams provides at-least-once delivery, so Lambda can receive the same stream record more than once during failover or scaling. Settings such as StartingPosition, BisectBatchOnFunctionError, batch size, and reserved concurrency affect batching and retry timing, but they do not eliminate duplicates.

DynamoDB Local does not automatically deduplicate stream records, and its stream emulator does not replicate managed-service failover or duplicate-injection timing. No confirmed documented difference between DynamoDB Local and the managed service specifically changes at-least-once duplicate delivery semantics. Treat local results as a development check, not production proof.

Steps for local development

1. Create a DynamoDB idempotency table with eventId as the partition key and an optional TTL attribute for marker expiration.
2. In the Lambda handler, extract eventID from each stream record.
3. Run a conditional put or transact write with attribute_not_exists(eventId) before the side effect.
4. If the conditional write fails, return success for that record without executing the side effect.
5. If the side effect writes to DynamoDB itself, combine the business change and the idempotency marker into one conditional update on the target item.

{
  "TableName": "IdempotencyMarkers",
  "Item": { "eventId": { "S": "RECORD_EVENT_ID" } },
  "ConditionExpression": "attribute_not_exists(eventId)"
}

Simulating duplicates repeatably

• Invoke the Lambda handler locally with the identical DynamoDB Streams event JSON two or more times and assert the side effect occurs exactly once.
• Write the same item twice to DynamoDB Local, capture both emitted stream records, and replay them to the handler in sequence.
• Avoid low-traffic tests with reserved concurrency; serialized retries can mask duplicate-processing bugs.

Verification

Locally, replay the same event JSON to the handler multiple times and confirm side effects are applied once. In a managed account, enable a DynamoDB stream, perform identical writes, and confirm the idempotency table rejects duplicate eventID values with a conditional check failure. Compare CloudWatch Logs from the managed stream against DynamoDB Local output before deploying to production.

Assumption: the Lambda code can be modified to include an idempotency check. If side effects are external API calls rather than DynamoDB writes, a separate idempotency table is required because a conditional update on the target item is not possible.

Missing diagnostic detail: Are the side effects writes to DynamoDB itself, or calls to external services or APIs? This determines whether a single conditional update on the target item is sufficient or whether a separate idempotency table is required.