Architecting Event-Driven Pipelines with Cosmos DB Change Feed

The Problem: Real-Time Reactivity Without Polling

Building a system that reacts to database changes often leads to inefficient polling patterns that waste Request Units (RUs) and introduce latency. Azure Cosmos DB's Change Feed solves this by providing a persistent, ordered log of inserts and updates. However, because it guarantees at-least-once delivery, a naive implementation will eventually cause data duplication or processing bottlenecks.

The critical takeaway: To build a production-ready pipeline, you must pair the Change Feed with an idempotent consumer and a dedicated lease container to track progress (checkpointing).

Requirements

• Cosmos DB Account: SQL (Core) API with a source container where ChangeFeedEnabled is true.
• Processor: An Azure Function with a Cosmos DB trigger or a custom application using the Change Feed Processor (CFP) library.
• Lease Store: A separate Cosmos DB container to store the state of the feed (checkpoints).
• Identity: A Managed Identity with Cosmos DB Built-in Data Reader on the source and Cosmos DB Built-in Data Contributor on the lease container.

The Smallest Suitable Design

The most efficient minimal design utilizes an Azure Function. This removes the need to manage the hosting infrastructure for the CFP library.

1. Container Configuration

Create a source container with a partition key that aligns with your event granularity (e.g., /tenantId). Run this via Azure CLI to establish the baseline:

az cosmosdb sql container create --account-name my-cosmos-acct --database-name event-db --name source-events --partition-key-path "/tenantId" --throughput 400

2. The Processor (Azure Function)

The function acts as the listener. It reads batches of changes and forwards them to a downstream service. The leaseCollectionName is where the function tracks which documents it has already processed.

{
  "bindings": [
    {
      "name": "documents",
      "type": "cosmosDBTrigger",
      "direction": "in",
      "leaseCollectionName": "leases",
      "connectionStringSetting": "CosmosDbConnectionString",
      "databaseName": "event-db",
      "collectionName": "source-events",
      "createLeaseCollectionIfNotExists": true
    }
  ]
}

3. Idempotent Consumer Logic

Because the processor may restart and re-read the last batch before a checkpoint was written, the downstream logic must be idempotent. This means processing the same event twice results in the same state as processing it once.

Trust and Data Boundaries

The Partition Key is the primary boundary. The Change Feed is processed per partition; if you use a low-cardinality key (e.g., /region), you limit the number of parallel processors that can work on the feed, potentially creating a bottleneck.

• Security Boundary: Use Azure RBAC to ensure the processor cannot delete data from the source container, only read it.
• Data Boundary: The lease container should be isolated from the source container to prevent RU contention between the business logic and the feed tracking.

Operational Checks

Monitoring a Change Feed is different from monitoring a standard API. You must track Lag.

Metric	Check Method	Warning Threshold
Feed Lag	Query lease container for LastUpdated timestamps.	> 5 minutes
Throttling	Monitor 429 errors in Application Insights.	Any sustained spike
RU Consumption	Azure Monitor: Total Request Units.	> 80% of provisioned

Failure Modes

• Duplicate Processing: Occurs when a processor crashes after executing logic but before updating the lease. Mitigation: Use a unique eventId in the document to deduplicate at the destination.
• Poison Pill Documents: A malformed document causes the function to crash repeatedly, stalling the entire partition. Mitigation: Implement a try-catch block that sends failing documents to a Dead Letter Queue (DLQ).
• Back-pressure: Downstream APIs slow down, causing the Change Feed to lag. Mitigation: Introduce an intermediate queue (e.g., Service Bus) between the function and the final destination.

Conditions for Redesign

The minimal Azure Function design should be replaced if:

• Exactly-Once Requirement: If the business cannot tolerate duplicates and cannot implement idempotency, you must move to a system with transactional outbox patterns.
• Extreme Throughput: If RU costs for the lease container become prohibitive, consider a custom CFP implementation on AKS for finer control over batch sizes.
• Multi-Region Requirements: If you need to process changes in the region where they occurred to avoid cross-region egress costs, deploy regional processor instances.

Verification Step

To verify the pipeline, perform the following sequence:

1. Insert a document into source-events and confirm the Azure Function log shows the event was processed.
2. Manually delete a lease entry in the leases container for that partition.
3. Observe the function re-processing the last batch of events, confirming the at-least-once behavior and verifying that your idempotency logic prevents duplicate side effects.