Using Cosmos DB Change Feed for Event‑Driven Workflows: A Practical Guide

Problem: How to react to every write in Cosmos DB without a separate CDC system

When you store data in Cosmos DB you often need to keep other services in sync—update a search index, send a notification, or feed a data lake. Pulling the entire container to detect changes is expensive and slow. Cosmos DB’s Change Feed gives you a built‑in, ordered log of every mutation. The key question: how do you consume it efficiently and reliably?

Thesis: The Change Feed is a first‑class event source, but you must manage leases, filtering, and multi‑region nuances to avoid duplicate or missed events.

1. Enabling the Feed (no extra cost)

Change Feed is enabled by default on every Core (SQL) container. For other APIs (MongoDB, Gremlin, Cassandra, Table) the same applies. The only thing you need to create is a lease container that holds the state of processed partitions.

# Azure CLI – create lease container (same throughput as source container is recommended)
az cosmosdb sql container create \
  --account-name myCosmosAccount \
  --database-name myDb \
  --name leases \
  --partition-key-path "/partitionKey" \
  --throughput 400

After the lease container exists, any consumer library will automatically write checkpoints here.

2. Consuming with the .NET Change Feed Processor

The SDK’s ChangeFeedProcessorBuilder handles leases, parallelism, and checkpointing. Below is a minimal example that writes each change to a log file.

using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.ChangeFeed;

var cosmosClient = new CosmosClient("AccountEndpoint=...;AccountKey=...;");
var container = cosmosClient.GetContainer("myDb", "orders");
var leaseContainer = cosmosClient.GetContainer("myDb", "leases");

var processor = container.GetChangeFeedProcessorBuilder("orderProcessor", async (IReadOnlyCollection changes, CancellationToken ct) =>
{
    foreach (var order in changes)
    {
        // Business logic – e.g., push to Event Hubs
        Console.WriteLine($"Order {order.Id} updated at {order._ts}");
    }
})
    .WithInstanceName("processor1")
    .WithLeaseContainer(leaseContainer)
    .WithLeaseRenewInterval(TimeSpan.FromSeconds(15))
    .WithLeaseDuration(TimeSpan.FromSeconds(30))
    .Build();

await processor.StartAsync();
// Application runs until cancelled

await processor.StopAsync();

Key points:

• Lease container holds a document per logical partition; each lease contains the last processed _lsn. The processor renews leases every 15 s and expires them after 30 s if not renewed.
• Checkpoints are written after each batch; if the process crashes you resume from the last checkpoint.
• Parallelism is controlled by WithInstanceName—multiple instances share the same lease container to scale out.

3. Filtering and “latest‑version‑only” mode

Change Feed returns every mutation in the order it happened per partition. Server‑side filtering is not available; you must filter client‑side. If you only care about the most recent state, you can skip intermediate updates by setting StartFromBeginning = false in the iterator. Note that deletes still surface as tombstone documents.

var iterator = container.GetChangeFeedIterator(ChangeFeedStartFrom.Now, ChangeFeedMode.Incremental);
while (iterator.HasMoreResults)
{
    foreach (var order in await iterator.ReadNextAsync())
    {
        // process only inserts/updates
        if (order._etag != null) { … }
    }
}

4. Multi‑Region Writes – Independent Feeds

In a multi‑region write account each region exposes its own Change Feed reflecting local writes. There is no global ordering across regions. If you need a single event stream, aggregate the feeds or add a logical timestamp property to each document and sort downstream.

5. Trade‑offs and Limitations

Aspect	Impact
Retention	Infinite until the document is deleted or container dropped. Great for replay but can grow large.
Throughput Cost	Reading the feed consumes RU/s from the source container (≈1–5 RU per 1 KB). Lease writes add a few hundred RU/s per container.
Partition Splits	When a partition splits, the processor may temporarily duplicate or miss changes if checkpoints are not atomic. The SDK handles this, but custom code must coordinate.
Delete Events	Deletes appear as tombstones with only system properties. If you need the pre‑delete payload, capture it before the delete or use a soft‑delete flag.
Filtering	No server‑side filter; client must discard unwanted changes.
Multi‑Region Ordering	No single global order; add logical timestamps if ordering is required.

6. Practical Checklist for Production

1. Create a dedicated lease container with throughput matching the source container.
2. Choose a consumer: Azure Functions trigger for low‑code scenarios, or SDK/Change Feed Processor for high control.
3. Implement idempotent processing (use _etag or _lsn to detect duplicates).
4. Monitor lease health: stale leases indicate a stuck consumer.
5. Set up diagnostics: enable DataPlaneRequests logs and filter by ReadChangeFeed to track RU consumption.
6. For multi‑region writes, decide whether to merge feeds or add a lastModified property for ordering.
7. Back up or replicate the lease container if you need to recover from a catastrophic failure.

Conclusion

Cosmos DB Change Feed turns your database into an event source without the overhead of an external CDC tool. By managing leases, filtering, and understanding multi‑region behavior, you can build reliable, low‑latency event‑driven pipelines that scale with your data.