Architecting Event-Driven Synchronization with Cosmos DB Change Feed

The Synchronization Problem

Maintaining data consistency across multiple microservices or creating read-optimized views often leads to "dual-write" problems, where an application attempts to update a primary database and a secondary index or message queue simultaneously. If one write fails, the system enters an inconsistent state.

The Change Feed in Azure Cosmos DB solves this by treating the database as the source of truth and the event emitter. It provides a persistent log of changes (inserts and updates) that external processors can consume asynchronously, ensuring that every change to a document is eventually propagated to downstream systems.

The Smallest Suitable Design

To implement a basic synchronization pipeline, you need three components: a source container, a lease container, and a Change Feed Processor (CFP) worker.

The Source and Lease Containers

The source container holds your operational data. The lease container is a specialized container used by the CFP library to store checkpoints. These checkpoints track which part of the change log each worker has processed, allowing the system to resume from the last known position after a crash or restart.

The Processor Logic

The CFP is a client-side library (available in .NET and Java SDKs) that manages the polling of the change feed. It handles the distribution of partitions across multiple worker instances to allow for horizontal scaling.

Configuration Example

Below is a conceptual configuration for a .NET worker. This logic should run in a scalable compute environment like Azure Functions or a Kubernetes pod.

// Required permissions: Cosmos DB Data Contributor on both containers
var leaseContainer = cosmosClient.GetContainer("database", "leases");
var sourceContainer = cosmosClient.GetContainer("database", "orders");

var processor = sourceContainer.GetChangeFeedProcessor("OrderSyncProcessor", 
    async (IReadOnlyCollection<Order> changes, CancellationToken cancellationToken) => 
    {
        foreach (var order in changes)
        {
            // Project data to a read-optimized Materialized View
            await SyncToReadModel(order);
        }
    }, 
    leaseContainer);

await processor.StartAsync();

Trust and Data Boundaries

The Change Feed operates within the boundary of a single Cosmos DB account. While the CFP can push data to external systems (like Azure Service Bus or another Cosmos DB container), the ordering guarantee only exists per partition key. Changes to documents with different partition keys may be processed out of order relative to each other.

Because the Change Feed is a "push-model" from the perspective of the SDK but a "pull-model" from the database, the trust boundary resides at the processor. The processor must be idempotent; if a worker crashes after processing a batch but before updating the lease, the next worker will process that same batch again.

Operational Checks and Failure Modes

Monitoring Processing Lag

The primary metric for health is Change Feed Lag. If the source container experiences a high-throughput burst, the CFP may fall behind. Check the lease container for high RU (Request Unit) consumption; if the lease container is throttled, the processor cannot checkpoint, leading to redundant processing and increased lag.

The Poison Pill Scenario

A "poison pill" occurs when a specific document triggers an unhandled exception in the processor logic. Because the CFP will retry the batch until it succeeds, a single malformed document can halt the entire partition's progress.

• Diagnostic: Monitor for a specific lease that never advances despite high CPU usage on the worker.
• Mitigation: Implement a try-catch block within the processor that sends failing documents to a separate "Dead Letter" container for manual review.

The Deletion Gap

By default, the Change Feed does not track deletes. If a document is hard-deleted from the source container, the downstream view will never know. To solve this, implement a Soft Delete pattern: add a boolean isDeleted property to the document and update it to true instead of deleting the record.

Design Evolution

The current design of a single CFP worker is sufficient for low-to-medium volumes. However, you should change the architecture if the following conditions occur:

Verification and Rollback

Verification: To verify the pipeline, perform a bulk update of 100 documents in the source container. Query the lease container to ensure the ContinuationToken is updating and verify that the downstream read-model reflects the changes within the expected latency window.

Rollback: Since the Change Feed is a read-only stream of the source container, there is no "rollback" of the feed itself. To reset the processor to a previous point in time, delete the entries in the lease container. This forces the CFP to start from the beginning of the feed (or the specified start time), effectively re-processing all historical changes.