Implementing Event-Driven Materialized Views with Cosmos DB Change Feed

The Problem: Read-Heavy Queries on Normalized Data

In Azure Cosmos DB, optimizing for write performance often means normalizing data or using a schema that doesn't align with every possible query pattern. When you need to support complex aggregations or read-optimized views without sacrificing write throughput, you face a trade-off: perform expensive cross-partition queries at runtime or maintain a separate, pre-computed view of the data.

The Change Feed solves this by providing a persistent record of changes to a container. It allows you to implement the CQRS (Command Query Responsibility Segregation) pattern, where the primary container handles writes and a secondary "materialized view" container handles optimized reads, kept in sync asynchronously.

How the Change Feed Mechanism Works

The Change Feed is a sorted list of documents that have been inserted or modified. It does not track the history of every single change to a document; rather, it provides the latest version of a document after a change has occurred. To process these changes at scale, the Change Feed Processor (CFP) library is used.

The CFP manages a Lease Container. This is a separate Cosmos DB container that acts as a state store, tracking which worker instance is processing which partition of the source data. This allows you to scale your processing logic across multiple compute instances (like Azure Functions or Kubernetes pods) without processing the same change twice.

Implementation Example: .NET SDK Configuration

To implement a materialized view, you need a source container (e.g., Orders) and a lease container (e.g., OrdersLeases). The following logic demonstrates how to initialize the processor to update a summary container.

// Required permissions: Cosmos DB Data Contributor or custom role with access to both containers
var leaseContainer = cosmosClient.GetContainer("DatabaseId", "OrdersLeases");
var sourceContainer = cosmosClient.GetContainer("DatabaseId", "Orders");

ChangeFeedProcessor processor = sourceContainer
    .GetChangeFeedProcessorBuilder("MaterializedViewWorker", HandleChangesAsync)
    .WithInstanceName("Worker-01")
    .WithLeaseContainer(leaseContainer)
    .Build();

await processor.StartAsync();

// The handler logic that updates the read-optimized view
async Task HandleChangesAsync(IReadOnlyCollection<Item> changes, CancellationToken cancellationToken)
{
    foreach (var item in changes)
    {
        // Example: Update a 'CustomerOrderSummary' container based on the Order change
        await UpdateMaterializedView(item);
    }
}

Operational Requirements and Risks

Critical Limitations and Common Mistakes

The Deletion Gap

A primary limitation of the Change Feed is that it does not capture deletions. If a document is hard-deleted from the source container, the Change Feed will not trigger. To solve this, implement Soft Deletes: add a boolean property (e.g., isDeleted: true) to the document. The Change Feed will see this update, allowing your handler to remove the corresponding entry from the materialized view.

Lack of 'Before' Images

The Change Feed provides the after image of the document. If your materialized view requires calculating a delta (e.g., subtracting a previous value from a new one), you must store the previous state in the materialized view itself or a separate state store to perform the comparison.

At-Least-Once Delivery

The CFP guarantees at-least-once delivery. In rare scenarios, such as a worker crashing immediately after processing a change but before updating the lease, a change may be processed twice. Your handler logic must be idempotent—meaning processing the same update multiple times results in the same final state.

Verifying the Implementation

To verify that the pipeline is functioning correctly, perform the following checks:

• Latency Check: Insert a document into the source container and measure the time it takes to appear in the materialized view.
• Scaling Check: Start a second instance of the processor with a different InstanceName. Observe the Lease container to ensure the partitions are redistributed between the two workers.
• Recovery Check: Stop the processor, make several updates to the source data, and restart the processor. Verify that the CFP resumes from the last recorded lease position and processes all missed changes.