Cosmos DB Change Feed Processor: Lease Management, Scaling, and Common Pitfalls

Why the Change Feed is a Game‑Changer

When a .NET service needs to react to every insert or update in a Cosmos DB container, a change feed provides a built‑in, ordered log. It guarantees that each partition’s operations surface in the order they were committed, and the SDK’s ChangeFeedProcessor library handles the heavy lifting of lease acquisition, checkpointing, and graceful shutdown. The result is a clean, event‑driven architecture without the overhead of a separate message broker.

1. The Lease Container – Your Partition‑Ownership Ledger

Every worker instance of the processor talks to a dedicated lease container. The lease document contains a leaseToken (the partition key value) and an ownerId. The SDK uses optimistic concurrency to ensure that only one worker owns a partition at any time.

• Throughput: Lease containers are read‑heavy. A rule of thumb is 400–1000 RU/s per 1000 partitions. Provision more if you expect >10 workers.
• Partition key: Use the same partitionKeyPath as the monitored container to keep leases co‑located with the data.
• TTL: Disable TTL on the lease container; leases must persist until the processor shuts down.

2. Scaling Patterns – From One Worker to a Cluster

The processor automatically balances partition ownership across workers that share the same processorName and lease container. When you add a new worker, the SDK performs a lease “steal” operation: it reads the current lease, applies a new ownerId, and writes it back if the document hasn’t changed. This pattern is safe but can be sensitive to RU starvation.

1. Launch multiple instances of your application (Docker, Kubernetes, Azure Functions).
2. All instances point to the same Cosmos DB account, lease container, and processor name.
3. When a worker starts, it polls the lease container at FeedPollDelay (default 5 s) to acquire new partitions.
4. Each worker processes its assigned partitions until the lease is renewed or the worker stops.

3. Tuning Latency vs. RU Consumption

The two knobs that most impact performance are FeedPollDelay and MaxItemsPerInvocation. Shorter delays reduce latency but increase read RU usage on the monitored container. Larger batches lower RU per item but increase processing time.

Parameter	Default	Typical Value for < 10k RUs
FeedPollDelay	5 s	2–3 s for low‑latency workloads
MaxItemsPerInvocation	100	200–500 for high‑throughput containers
MaxItemCountPerPage	1000	1000–2000 for large documents

4. Edge Cases – Deletes, Soft‑Delete, and Analytical Store

The change feed does not emit delete operations by default. If your downstream system relies on deletes, you have three options:

• Soft‑Delete pattern: Add an isDeleted flag and TTL. The feed will still emit the full document.
• Analytical Store (Synapse Link): Enables delete capture but introduces a 2–5 minute latency and requires a separate Spark job to read.
• Custom CDC layer: Write a trigger that logs deletes to a separate container and consume that with the processor.

Remember that partial updates via the Patch API appear as full document replacements in the feed; consumers must compute deltas if needed.

5. Worked Example – A Minimal .NET Processor

The following code shows how to wire up a processor that writes each change to a log file. Run it from a console app; each instance can be started in a separate container.

using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.ChangeFeedProcessor;

var cosmosClient = new CosmosClient("AccountEndpoint=https://<account>.documents.azure.com:443/;AccountKey=<key>;");
var monitoredContainer = cosmosClient.GetContainer("database", "orders");
var leaseContainer = cosmosClient.GetContainer("database", "orders_leases");

var processor = new ChangeFeedProcessorBuilder()
    .WithHostName("orders-processor-host")
    .WithFeedContainer(monitoredContainer)
    .WithLeaseContainer(leaseContainer)
    .WithProcessorName("orders-processor")
    .WithChangeFeedProcessorOptions(new ChangeFeedProcessorOptions
    {
        FeedPollDelay = TimeSpan.FromSeconds(3),
        MaxItemsPerInvocation = 200
    })
    .WithEventHandler((changes, cancellationToken) =>
    {
        foreach (var doc in changes)
        {
            Console.WriteLine($"{DateTime.UtcNow:o} – Updated {doc.Id}");
        }
        return Task.CompletedTask;
    })
    .Build();

await processor.StartAsync();
Console.WriteLine("Processor started. Press any key to stop.");
Console.ReadKey();
await processor.StopAsync();

Key points:

• All instances must share the same ProcessorName and lease container.
• Use WithHostName to differentiate instances; it is part of the lease document.
• Checkpointing is automatic; the SDK writes a ContinuationToken after each successful batch.

6. Trade‑offs & Limitations

• Delete Capture: If you need deletes, the base change feed is insufficient. Soft‑delete or Synapse Link adds complexity.
• Cross‑Region Reads: In a multi‑master account, each region hosts its own change feed. Reading from a single region will miss writes from others unless you aggregate feeds.
• Ordering Guarantees: Ordering is only within a partition. If you need global ordering, you must add a sequencing field.
• Lease RU Pressure: Under‑provisioned lease containers cause lease acquisition failures, leading to duplicate processing. Monitor RU/s on the lease container and scale as needed.

7. Actionable Checklist Before Production

1. Enable change feed on the target container and verify that EnableChangeFeed is set to true.
2. Provision a lease container with at least 400 RU/s per 1000 partitions; set partitionKeyPath to the same as the monitored container.
3. Run a single processor instance and confirm that leases appear in the lease container.
4. Add a second instance; verify that partition ownership is redistributed.
5. Monitor FeedPollDelay and MaxItemsPerInvocation to balance latency and RU consumption.
6. If deletes are required, implement a soft‑delete flag or enable Synapse Link and test delete capture.
7. For multi‑region accounts, decide whether to read feeds from all regions or aggregate them; configure your processors accordingly.
8. Set up alerts on lease container RU/s and on processor failures to detect duplicate processing.
9. Document the processor name and lease container so that new deployments reuse the same configuration.
10. Periodically run a replay using ChangeFeedStartFrom.RequestTime to ensure idempotent processing.

By following these steps, you can harness Cosmos DB’s change feed as a reliable, low‑latency event source while avoiding the most common pitfalls.