Solving Downstream Latency with Cosmos DB Change Feed

The Polling Problem

Many architectures rely on a poll-and-process pattern to keep downstream systems in sync. A background worker queries a database every few minutes for records where last_modified > last_check_time. This approach creates a frustrating trade-off: you either accept high latency with updates taking minutes to propagate, or you overwhelm your database with constant expensive queries that consume Request Units without adding business value.

The solution is to move from a pull-based model to a push-based event stream. Azure Cosmos DB Change Feed provides a persistent, append-only log of all inserts and updates within a container, allowing you to trigger downstream actions the moment data changes.

How Change Feed Operates

Change Feed is a built-in capability of the Cosmos DB engine, not a separate service. It tracks changes at the partition level, ensuring that all modifications to a specific partition key are recorded in the exact order they occurred. This is critical for maintaining data integrity in downstream caches or search indexes.

When a document is inserted or updated, the change is appended to the feed. The feed is primarily designed for inserts and updates, with delete tracking available via specific configurations. Because the feed is distributed across physical partitions, it scales horizontally along with your database throughput.

Implementing a Real-Time Pipeline

To use the Change Feed you enable the feature on the container and implement a Change Feed Processor. The processor is a library provided in the SDKs that manages distributing work across multiple worker instances and tracking checkpoints, markers that tell the processor where it left off in the stream.

Example: Streaming Order Updates to Kafka

Consider an e-commerce platform that needs to send order status updates to a Kafka topic for the shipping department. Instead of polling the Orders container, a Change Feed Processor acts as the bridge.

// Conceptual .NET implementation using the Change Feed Processor
Container leaseContainer = client.GetContainer('Database', 'leases');
Container sourceContainer = client.GetContainer('Database', 'Orders');

ChangeFeedProcessor processor = sourceContainer
    .GetChangeFeedProcessorBuilder('OrderStreamingProcessor', HandleChangesAsync)
    .WithInstanceName('Worker-01')
    .WithLeaseContainer(leaseContainer)
    .Build();

await processor.StartAsync();

async Task HandleChangesAsync(IReadOnlyCollection<Order> changes, CancellationToken cancellationToken)
{
    foreach (var order in changes)
    {
        await kafkaProducer.ProduceAsync('order-updates', new Message { Key = order.Id, Value = order });
    }
}

Run this code within an Azure Function or a containerized microservice. The leaseContainer is a separate Cosmos DB container used to store processor state. It ensures that if you scale to multiple worker instances, each instance handles a unique subset of partitions without duplicating events.

Engineering Trade-offs and Constraints

While Change Feed eliminates polling, it introduces specific operational requirements:

• RU Consumption: Reading from the Change Feed consumes RUs. If your container has high write volume, the read activity of the processor can contribute to RU throttling. Provision throughput that accounts for both writes and feed reads.
• Partition-Level Ordering: Ordering is guaranteed per partition key. If business logic requires a strict global sequence across the entire database, you must implement sequencing downstream, as the feed delivers changes in parallel across partitions.
• Retention Windows: The feed is not a permanent archive. If a consumer crashes and stays offline longer than the retention window, typically 24 hours to 7 days depending on API, the processor may miss events. In that case you must trigger a full re-scan to re-seed the downstream system.

Verifying the Integration

To confirm the pipeline is functioning, use Azure Monitor to track Change Feed Read Count and Change Feed Write Count metrics. A healthy system shows correlation between write spikes and subsequent read spikes. To test lag, introduce a controlled delay in HandleChangesAsync and monitor the lease container to see how the processor handles the backlog.