Real‑Time Event Streams with Cosmos DB Change Feed: From SQL API to Azure Functions

Concrete Problem

Your application writes data to a Cosmos DB SQL container, but downstream services (analytics, notifications, inventory sync) need to react instantly to every insert or update. Polling the container or writing custom triggers adds latency and complexity. You need a reliable, server‑side log that delivers changes in the order they were committed.

Thesis

Azure Cosmos DB’s Change Feed is a built‑in, read‑only stream that emits every insert or update to a container in commit order. By enabling it on a container and consuming it with the Change Feed Processor library, you can build a real‑time, event‑driven architecture without affecting write latency or throughput.

Step 1 – Enable the Change Feed on a Container

Change Feed is enabled per container via the enableChangeFeed property. You can set it in the Azure portal, an ARM template, or programmatically.

{
  "type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers",
  "apiVersion": "2024-02-15",
  "name": "myAccount/myDatabase/myContainer",
  "properties": {
    "resource": {
      "id": "myContainer",
      "offerThroughput": 400,
      "partitionKey": {
        "paths": ["/category"],
        "kind": "Hash"
      },
      "enableChangeFeed": true,
      "changeFeedPolicy": {
        "maxItemCount": 500,
        "timeBetweenPollsInSeconds": 5
      }
    }
  }
}

Alternatively, via the SDK:

CosmosContainerProperties props = new CosmosContainerProperties("myContainer", "/category");
props.setEnableChangeFeed(true);
props.setChangeFeedPolicy(new CosmosChangeFeedPolicy(500, 5));
client.createContainer(props, 400, ThroughputProperties.createManualThroughput(400));

After enabling, every write appears in the feed within seconds. The feed is read‑only, so write latency stays unchanged.

Step 2 – Consume with the Change Feed Processor

The Change Feed Processor (CFP) library handles partition leasing, scaling, and at‑least‑once delivery. Below is a minimal Java example that logs each change to the console.

public static void main(String[] args) throws Exception {
    String endpoint = "";
    String key = "";
    String database = "myDatabase";
    String container = "myContainer";

    CosmosClient client = new CosmosClientBuilder()
        .endpoint(endpoint)
        .key(key)
        .consistencyLevel(ConsistencyLevel.SESSION)
        .buildClient();

    CosmosContainer cosmosContainer = client.getDatabase(database).getContainer(container);

    ChangeFeedProcessor changeFeedProcessor = new ChangeFeedProcessorBuilder()
        .hostName("processor-host-1")
        .feedContainer(cosmosContainer)
        .leaseContainer(client.getDatabase(database).getContainer("leases"))
        .processItem((item, context) -> {
            System.out.println("Change: " + item.getId() + " at " + Instant.now());
        })
        .buildChangeFeedProcessor();

    changeFeedProcessor.start();
    System.out.println("Processor started – press Enter to stop.");
    System.in.read();
    changeFeedProcessor.stop();
}

Key points:

• Lease Container – a dedicated container that stores lease information. Create it with at least 400 RU/s.
• CFP guarantees at‑least‑once delivery. Your code should deduplicate if the same document is processed more than once.
• Ordering is only guaranteed within a single logical partition (e.g., /category).

Step 3 – Wire to Azure Functions (Optional)

For serverless pipelines, replace the console logger with an Azure Function trigger. The function receives a CosmosDBTrigger binding that streams changes directly.

[FunctionName("ProcessChangeFeed")]
public static void Run(
    [CosmosDBTrigger(
        databaseName: "myDatabase",
        collectionName: "myContainer",
        ConnectionStringSetting = "CosmosDBConnection",
        LeaseCollectionName = "leases",
        CreateLeaseCollectionIfNotExists = true
    )]IReadOnlyList<Document> input,
    ILogger log)
{
    if (input != null && input.Count > 0)
    {
        foreach (var doc in input)
        {
            log.LogInformation($"Processed document {doc.Id} at {DateTime.UtcNow}");
            // Forward to Event Hubs, Kafka, or trigger business logic
        }
    }
}

Azure Functions automatically scales with the number of partitions and handles lease contention.

Trade‑offs & Limitations

Aspect	What to Watch For
Retention	Changes are kept as long as the container’s TTL is enabled. Disable TTL to keep changes indefinitely, but storage costs rise.
Idempotency	At‑least‑once semantics mean the same change can be delivered twice. Store a hash (e.g., _etag) or use a message queue that de‑duplicates.
Ordering	Guaranteed only within a single logical partition. Cross‑partition ordering is not preserved.
Throughput Impact	Reading the feed does not affect write RU consumption, but heavy consumption can throttle the processor if the container’s RU budget is low.

Actionable Next Steps

1. Enable enableChangeFeed on your target container.
2. Deploy a Change Feed Processor (Java, .NET, Node, or Python) or an Azure Function with a Cosmos DB trigger.
3. Implement deduplication logic if you scale across multiple processor instances.
4. Monitor latency by timestamping writes and receipt of change events; adjust maxItemCount or throughput as needed.
5. Consider TTL settings to balance cost and retention.

Diagram