Replacing Polling with Cosmos DB Change Feed: A Practical Migration Path

The polling problem

Teams that synchronize data from Cosmos DB often start with a simple pattern: run a scheduled query, compare results against a local cache, and apply differences downstream. This works at low volume but creates three compounding problems as throughput grows. First, each poll consumes request units (RU/s) whether or not data changed. Second, the polling interval becomes a latency floor — if you poll every five minutes, downstream systems are always at least five minutes stale. Third, the query logic grows complex when you need to detect updates, inserts, and deletes across partitions.

Ordering guarantees and partition reality

The change feed guarantees ordering within a single partition key. If your container uses /customerId as the partition key, all changes for customer A appear in sequence, and all changes for customer B appear in sequence, but the interleaving between A and B is not defined. Any consumer that merges partitions must tolerate out-of-order delivery across keys. This is not a bug — it is the direct consequence of Cosmos DB's horizontal scaling model. Design your downstream processing to be partition-aware or to reorder using event timestamps if global ordering matters.

Scaling with the Change Feed Processor

The ChangeFeedProcessor library (or the Azure Functions Cosmos DB trigger) handles the operational complexity of distributing partitions across multiple consumer instances. It uses a lease container to track which instance owns which partition range and where each instance has checkpointed. When you add a second processor instance, the lease container rebalances ownership automatically — no manual partition assignment required. The lease container consumes its own RU/s; monitor it separately because a hot lease container can become a bottleneck before the data container does.

Worked example: orders to reports

The following C# snippet shows a minimal processor that reads an orders container and upserts a denormalized summary into a reports container. It assumes .NET 8, the Microsoft.Azure.Cosmos SDK v3.35+, and a lease container named leases.

using Microsoft.Azure.Cosmos;
using System.Text.Json;

var cosmosClient = new CosmosClient("", "");
var ordersContainer = cosmosClient.GetContainer("retail", "orders");
var reportsContainer = cosmosClient.GetContainer("retail", "reports");
var leaseContainer = cosmosClient.GetContainer("retail", "leases");

var processor = ordersContainer.GetChangeFeedProcessorBuilder<OrderDocument>(
    processorName: "orders-to-reports",
    onChangesDelegate: async (IReadOnlyCollection<OrderDocument> changes, CancellationToken ct) =>
    {
        foreach (var order in changes)
        {
            var summary = new OrderSummary
            {
                Id = order.Id,
                PartitionKey = order.CustomerId,
                CustomerId = order.CustomerId,
                TotalAmount = order.Items.Sum(i => i.Price * i.Quantity),
                LastUpdated = order.UpdatedAt,
                Status = order.Status
            };
            await reportsContainer.UpsertItemAsync(summary, new PartitionKey(order.CustomerId), cancellationToken: ct);
        }
    })
    .WithInstanceName(Environment.MachineName)
    .WithLeaseContainer(leaseContainer)
    .Build();

await processor.StartAsync();
Console.WriteLine("Processor started. Press Enter to stop.");
Console.ReadLine();
await processor.StopAsync();

OrderDocument and OrderSummary are plain C# classes with Id, CustomerId, and other properties matching your schema. The handler is idempotent because UpsertItemAsync overwrites the same Id + partition key combination. This matters: the change feed provides at-least-once delivery, so the same document can appear multiple times after a failure and restart.

The delete blind spot

The change feed does not emit events for document deletions. If your downstream system must react to deletes — for example, removing a report when an order is cancelled — you have two practical options. Implement a soft delete by adding an isDeleted flag (or status: "cancelled") and updating the document instead of deleting it. The update appears in the feed and your handler can propagate the tombstone. Alternatively, maintain a separate tombstone collection where you write a minimal record ({ id, partitionKey, deletedAt }) before deleting the original document. Both approaches add write overhead but are the only way to make deletes visible to change feed consumers.

Checkpointing and idempotency

The processor checkpoints after each batch completes successfully. If your handler crashes mid-batch, the next run restarts from the last checkpoint and reprocesses the same documents. This is why the example uses UpsertItemAsync — reprocessing produces the same result. If your downstream operation is not naturally idempotent (e.g., sending an email, incrementing a counter), add a deduplication layer: store the processed change feed _lsn (log sequence number) or document _etag in a side table and skip duplicates.

Verify before you migrate

Before cutting over from polling, run a shadow validation: create a test container, enable the change feed, and run the processor locally against a copy of production data. Insert and update documents, then confirm events arrive in partition order. Delete a document and verify no event appears — this validates your soft-delete design. Scale to two processor instances and inspect the lease container to see partition distribution and checkpoint advancement. Only after these checks should you retire the polling job.

Closing thought

Moving from polling to the change feed shifts your architecture from pull to push, eliminates wasted RU/s on empty queries, and reduces end-to-end latency to the propagation delay of the feed itself (typically sub-second). The trade-offs — no delete events, partition-local ordering, lease container monitoring — are explicit and manageable. Start with one high-volume container, instrument the lease throughput, and expand once the pattern is proven.