Using Cosmos DB Change Feed to Build Materialized Views Without a Sync Job

You have a Cosmos DB container partitioned by customerId because that's how 90% of your queries run. Then the product team asks for a view "by order status across all customers," and suddenly every query is a cross-partition fan-out that burns RU/s and latency. The usual answers — a nightly ETL job or dual-writes from the application — either lag behind reality or make every write path fragile. The Change Feed gives you a third option: let the database itself stream every insert and update, and project that stream into a second container partitioned the way the new query pattern needs.

What the Change Feed actually gives you

Every Cosmos DB container keeps a persistent, ordered log of inserts and updates, available per logical partition. You don't enable it or pay extra for its existence — it's just there. What you build on top of it is a consumer. The important guarantees to design around:

• Order within a partition key: changes for a given item appear in the order they happened. There is no ordering guarantee across partition keys.
• At-least-once delivery: your consumer can see the same change twice, especially after retries or worker rebalancing. Idempotency is your job, not the feed's.
• Latest state only: the feed exposes the current version of an item, not a history. If an item changes three times between polls, you may only see the final version.
• No deletes: deletes never appear. If you need deletion propagation, use a soft-delete flag (an isDeleted property plus a TTL to eventually purge), and treat that flag as a normal update in your consumer.

Why the Change Feed Processor matters more than the feed

Reading the feed raw means managing continuation tokens per partition range, and partition ranges can split or merge as your data grows. Doing that by hand is a trap. The Change Feed Processor (CFP) library — built into the .NET and Java SDKs, and wrapped by the Azure Functions Cosmos DB trigger — handles the unglamorous parts:

• Leases: a separate "leases" container tracks which worker owns which partition range and how far it has read (the checkpoint).
• Load balancing: add a second worker instance and the lease ownership rebalances automatically; kill one and the survivor picks up its ranges from the last checkpoint.
• Checkpointing: progress is persisted so a restart resumes instead of replaying everything.

Two costs people miss: the lease container consumes its own RU/s (small, but nonzero — autoscale or a shared throughput database works fine), and checkpoint frequency is a trade-off. Checkpoint after every batch and you pay write RUs constantly; checkpoint rarely and a crash replays more duplicates onto your consumer.

Worked example: projecting orders into a by-status view

Assume a source container orders partitioned by /customerId, and a target container ordersByStatus partitioned by /status. An Azure Functions trigger (C#, isolated worker, Functions v4 with the Cosmos extension) looks like this:

[Function("ProjectOrdersByStatus")]
public async Task Run(
    [CosmosDBTrigger(
        databaseName: "sales",
        containerName: "orders",
        Connection = "CosmosConnection",
        LeaseContainerName = "leases",
        CreateLeaseContainerIfNotExists = true)]
    IReadOnlyList<Order> changes,
    [CosmosDBInput(
        databaseName: "sales",
        containerName: "ordersByStatus",
        Connection = "CosmosConnection")]
    CosmosClient client)
{
    var target = client.GetContainer("sales", "ordersByStatus");
    foreach (var order in changes)
    {
        if (order.IsDeleted)
        {
            await target.DeleteItemAsync<Order>(
                order.Id, new PartitionKey(order.Status));
            continue;
        }
        // Upsert keyed by order id: replaying the same change is a no-op.
        await target.UpsertItemAsync(order, new PartitionKey(order.Status));
    }
}

The idempotency here comes from using the source item's id as the target item's id and upserting: processing the same change five times produces the same target document. If your projection creates derived documents (e.g., aggregates), you need a stronger scheme — store the source item's _etag or a version field in the target and skip older versions.

Run this in the Azure portal or via func start locally with the Functions Core Tools; the identity or connection string in CosmosConnection needs read/write on both containers plus the leases container. To verify: insert an order into orders, then query ordersByStatus with WHERE c.status = 'pending' and confirm the document appears within a few seconds. Then scale the Function to two instances (or stop one) and confirm processing continues — that's the lease mechanism doing its job. One caveat: the example above doesn't handle an order changing status, which would leave a stale document under the old partition key. In practice, include a deterministic target id and delete-by-lookup, or store a status history — design for this before you ship.

The trade-offs worth saying out loud

Lag is a first-class metric. If your consumer processes changes slower than the write rate, the projection drifts behind. Monitor the estimated lag (the CFP exposes it; the Functions trigger surfaces it through metrics) and alert on it. A materialized view that's silently ten minutes stale is worse than no view.

It's not a transaction. The source write succeeds and the projection happens later. Any read of the target can observe a state the source has already moved past. If a workflow needs read-your-writes on the projected view, this pattern is the wrong tool.

Backfills need a separate path. The feed doesn't replay history from before your consumer existed. For initial population, run a one-time query over the source and write through the same projection logic, then start the processor from "now" (or use the all-versions-and-deletes mode in preview/SDK variants where available — verify current support before relying on it).

Where to start

Pick one query pattern that's currently cross-partition and painful. Stand up a second container with the right partition key, wire a minimal CFP consumer (the Functions trigger is the fastest route), and measure two things for a week: end-to-end lag under peak write load, and duplicate-delivery frequency during a forced restart. If both look acceptable, you have a pattern you can repeat for every new read model — without ever touching the write path again.