Using Azure Cosmos DB Change Feed to Build Materialized Views for Multi‑Dimensional Queries

Problem: Queries Slow When You Need More Than One Partition Key

Azure Cosmos DB lets you pick a single partition key per container. When your application must query the same data by different attributes (e.g., look up orders by customer ID and> by date), a query that does not match the partition key becomes a cross‑partition scan. Those scans consume more Request Units (RUs) and add latency, especially as data volume grows.

Thesis: Use the Change Feed to Keep Synchronized Copies Optimized for Each Query Pattern

The Change Feed provides an ordered, reliable stream of inserts and updates. By projecting that stream into additional containers—each partitioned on a key that matches a common query pattern—you create materialized views that can be read with low‑cost, single‑partition queries. The Change Feed Processor (CFP) library handles checkpointing and distributes the work across compute instances, so the pattern scales.

How the Pattern Works

1. Source container holds the canonical data, partitioned by the key that matches your primary write workload.
2. A Change Feed Processor (hosted in an Azure Function, VM, or Kubernetes pod) reads the feed, receives batches of changed documents, and writes them to one or more destination containers (the materialized views).
3. The CFP uses a lease container to store checkpoints, enabling multiple instances to share the workload without losing progress.
4. Read operations target the destination container whose partition key aligns with the query, avoiding cross‑partition scans.

Worked Example: .NET Change Feed Processor

The following snippet shows a minimal host that projects changes from orders (partitioned by orderId) into two views: ordersByCustomer (partitioned by customerId) and ordersByDate (partitioned by orderDate). Replace placeholders with your actual values.

using System;
using System.Threading.Tasks;
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.ChangeFeed;
using Microsoft.Azure.Cosmos.ChangeFeed.Processor;

class Program
{
    static async Task Main()
    {
        // Source container (writes happen here)
        var sourceClient = new CosmosClient("<SOURCE_ACCOUNT_ENDPOINT>", "<SOURCE_KEY>");
        var sourceContainer = sourceClient.GetContainer("database", "orders");

        // Destination containers (views)
        var destClient = new CosmosClient("<DEST_ACCOUNT_ENDPOINT>", "<DEST_KEY>");
        var ordersByCustomer = destClient.GetContainer("database", "ordersByCustomer");
        var ordersByDate = destClient.GetContainer("database", "ordersByDate");

        // Lease container for checkpointing
        var leaseContainer = destClient.GetContainer("database", "leases");

        // Build the processor
        var processor = sourceContainer.GetChangeFeedProcessorBuilder("ordersProcessor", async (changes, cancellationToken) =>
        {
            foreach (var item in changes)
            {
                // Deserialize to your domain type if needed
                var order = (Order)item;

                // Write to each view
                await ordersByCustomer.CreateItemAsync(order, new PartitionKey(order.CustomerId));
                await ordersByDate.CreateItemAsync(order, new PartitionKey(order.OrderDate.ToString("yyyy-MM-dd")));
            }
        })
        .WithInstanceName(Environment.MachineName)
        .WithLeaseContainer(leaseContainer)
        .Build();

        await processor.StartAsync();
        Console.WriteLine("Processor started. Press ENTER to stop.");
        Console.ReadLine();
        await processor.StopAsync();
    }
}

public class Order
{
    public string Id { get; set; }
    public string CustomerId { get; set; }
    public DateTime OrderDate { get; set; }
    // other properties...
}

Where to run: Deploy the compiled executable to an Azure Function (timer‑triggered or always‑on), a VM, or a Kubernetes pod. The identity executing the code needs Read permission on the source container and Create/Replace on the destination containers and lease container. Grant these via Azure RBAC (e.g., Cosmos DB Built-in Data Contributor) or via resource tokens.

Expected checks: After starting the processor, monitor the Change Feed Processor logs for batches processed. In the Azure Portal, navigate to Metrics → Request Units for each destination container; you should see a steady RU consumption that mirrors the write rate on the source. To verify correctness, query a known order ID in ordersByCustomer and confirm the returned document matches the source.

Trade‑off and Limitations

• Storage cost: Data is duplicated; each view consumes storage proportional to the source size.
• Write RU increase: Every source write incurs additional RUs for each destination container (source write + N view writes). Estimate the total as source_RUs × (1 + number_of_views).
• Eventual consistency: There is a latency gap (typically seconds to a few minutes) between a write to the source and its appearance in a view. Applications requiring strong consistency must read directly from the source or implement a read‑through pattern.
• Delete handling: Hard deletes do not appear in the Change Feed. Use a soft‑delete flag (e.g., IsDeleted) or a tombstone document if you need deletions reflected in views.

Practical way to check the result: Run a cross‑partition query on the source (e.g., SELECT * FROM orders WHERE customerId = 'C123' AND orderDate = '2026-09-01') and note the RU charge. Then run the equivalent query on the appropriate view (e.g., SELECT * FROM ordersByCustomer WHERE customerId = 'C123' AND orderDate = '2026-09-01'). The view query should show a significantly lower RU cost because it is a single‑partition request.

Actionable Closing

If your workload suffers from costly cross‑partition queries, consider the Change Feed‑driven materialized view pattern:

1. Identify the most frequent query dimensions and create a destination container for each, partitioned on the relevant key.
2. Deploy a Change Feed Processor host with lease container, ensuring proper RBAC.
3. Monitor storage and RU metrics; adjust the number of views or consolidate if cost outweighs benefit.
4. For delete‑heavy scenarios, implement a soft‑delete flag and update the processor to propagate the flag to views.

By keeping the source container optimized for writes and the views optimized for reads, you achieve predictable, low‑latency access patterns without sacrificing scalability.