Architecting Event-Driven Processing with Cosmos DB Change Feed and Azure Functions

The Challenge of Polling vs. Streaming Events

Many developers attempt to react to data changes in Cosmos DB by polling the container, which introduces unnecessary latency and consumes Request Units (RUs) even when no data changes. The better alternative is the Cosmos DB Change Feed, which allows applications to watch changes like inserts, updates, and deletes in near real-time. However, moving from polling to a trigger-based architecture requires a shift in how you handle state, scaling, and failure recovery to ensure your system remains consistent without losing data events.

The Smallest Viable Design

To build a functional event-driven pipeline, the minimal architecture consists of a single Cosmos DB container and an Azure Function App. The configuration requires:

• Container: Use a synthetic partition key (e.g., tenantId or orderId) to ensure related events are grouped.
• Function: Deployed on a Consumption or Premium plan, utilizing the Cosmos DB trigger.
• Lease Container: A separate container used by the trigger to store "continuation tokens," which track where the function left off in the stream.

For starters, set batchSize=100 and maxItemsPerInvocation=100. This balances throughput with execution time limits.

Trust Boundaries and Data Security

In this architecture, the trust boundary exists between the database and the compute layer. The Azure Function should use a Managed Identity granted only the Cosmos DB Built-in Data Contributor role. This avoids hardcoding connection strings with master keys.

The lease container is internal state, not business data. While it stores sequence numbers, it should not be queried by external services. If your security requirements are strict, ensure the Function uses VNet integration and connects to Cosmos DB via a private endpoint to prevent payloads from traversing the public internet.

Operational Checks and Monitoring

Because the Change Feed is asynchronous, standard "success/fail" metrics are insufficient. You must monitor the following in Application Insights:

• Function Lag: Calculate the difference between the maximum Log Sequence Number (LSN) in the feed and the current LSN in the lease container. High lag indicates your consumer cannot keep up.
• Throttled Requests: If the lease container's RU consumption exceeds 70% of provisioned capacity, the entire stream will stall.
• Execution Duration: Long-running batches can cause timeouts, leading to re-processing.

To verify health manually, run a diagnostic query on the lease container: SELECT count(*) FROM c WHERE _self = 'lease' to detect if specific partitions are stuck or orphaned.

Failure Modes and Recovery

The Change Feed guarantees at-least-once delivery, which means your system must handle specific scenarios:

• Function Crash mid-batch: If the function fails before checkpointing, it will re-process the entire batch upon restart. Ensure downstream logic is idempotent.
• Poison Messages: If a specific document causes a deserialization error, the partition will stall indefinitely. Implement a try/catch block within your code to move failing messages to a dead-letter queue (like Azure Storage Queues).
• Partition Splits: When Cosmos DB splits a partition, the trigger automatically creates new leases. This may cause brief duplicate processing during the migration period.

When to Change the Design

The minimal design above needs modification if your requirements shift toward:

• Strict Exactly-Once: If downstream systems cannot handle duplicates, implement the transactional outbox pattern where you write the event and a status marker in the same Cosmos DB transaction.
• High Throughput (>10k RU/s): Move from the Consumption plan to the Premium plan with pre-warmed instances to avoid cold starts and increase batchSize.
• Global Distribution: If you need low-latency consumers globally, deploy a Function in each region and use a regional lease prefix to minimize cross-region latency.