Cosmos DB Change Feed for Microservice Event Streaming

Problem: Needing a Simple Event Stream

When building microservices on Azure, teams often reach for Service Bus, Event Hubs, or Kafka to propagate state changes. Adding another managed service introduces operational overhead, cost, and latency considerations. If your data already lives in Azure Cosmos DB, you might wonder whether the database itself can push changes to consumers without a separate broker.

Thesis: Change Feed as a Built‑In Event Log

Azure Cosmos DB Change Feed offers a persistent, ordered log of inserts and updates for each partition. It scales with the container’s provisioned throughput, provides lease‑based checkpoints for multiple consumers, and can be consumed directly via SDK processors or Azure Functions triggers. This makes it a viable lightweight event stream for many microservice scenarios.

How Change Feed Works

Each container maintains an internal log of changes. A Change Feed processor reads this log, groups changes by partition key range, and uses a lease collection (another Cosmos DB container) to store checkpoints. Multiple instances of the processor can run concurrently; each lease guarantees that a given partition range is processed by exactly one consumer at a time.

Supported consumers include:

• Azure Functions with the Cosmos DB trigger
• Custom .NET/Java SDK processor
• Spark connector
• Kafka‑compatible sink (via Azure Cosmos DB Kafka connector)

Worked Example: Azure Function Trigger

The following example shows a minimal Azure Function written in C# that logs each change. It assumes you have a Cosmos DB account, a database named OrdersDb, and a container Orders with partition key /customerId. The function will be triggered for every insert or update.

using System;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.Documents;

public static class OrderChangeFunction
{
    [FunctionName("OrderChangeFunction")]
    public static void Run([CosmosDBTrigger(
        databaseName: "OrdersDb",
        collectionName: "Orders",
        ConnectionStringSetting = "CosmosDbConnection",
        LeaseCollectionName = "leases")] IReadOnlyList changes,
        ILogger log)
    {
        if (changes == null || changes.Count == 0)
        {
            return;
        }

        foreach (var doc in changes)
        {
            log.LogInformation($"Change received: id={doc.Id}, customerId={doc.GetPropertyValue("customerId")}");
        }
    }
}

To run this locally you need the Cosmos DB Emulator. Set the app setting CosmosDbConnection to the emulator’s connection string:

AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==;

After deploying the function (or pressing F5 in Visual Studio), insert a document into the Orders container via the emulator’s Data Explorer or SDK:

var doc = new { id = Guid.NewGuid().ToString(), customerId = "cust-123", amount = 99.99 };
await container.CreateItemAsync(doc, new PartitionKey("cust-123"));

You should see a log line in the function output similar to:

Change received: id=..., customerId=cust-123

Checkpoint information is stored in the leases container; stopping and restarting the function will resume from the last processed change.

Trade‑offs and Limitations

• Change Feed only captures inserts and updates. To react to deletes you must enable the optional delete feed or implement a soft‑delete pattern (e.g., add a deleted flag).
• Each read of the Change Feed consumes request units proportional to the payload size; very high write rates can increase RU consumption noticeably.
• Latency is typically sub‑second for provisioned throughput containers, but autoscale modes may add a few seconds during scale‑up/down events.
• Because the feed is tied to a single container, fan‑out to multiple unrelated topics requires either multiple containers or post‑processing routing inside the consumer.

Practical way to verify the behavior: run the emulator, create the container, insert a few documents, and confirm that the Function logs each change and that the lease collection shows updated continuation tokens after each batch.

Actionable Closing

If your microservice architecture already stores state in Azure Cosmos DB, consider using Change Feed as the first‑class event stream before adding a separate message broker. Start with a small Azure Function or SDK processor, monitor RU usage via the Cosmos DB metrics blade, and add a soft‑delete field if you need delete events. This approach reduces operational complexity while preserving low‑latency, ordered delivery of changes.