Using Azure Cosmos DB Change Feed with Azure Functions: Checkpointing Strategies and Trade‑offs

Problem: Missing updates when polling Cosmos DB

Many applications still poll a Cosmos DB container for new or changed documents, which wastes request units, adds latency, and can miss updates if the polling interval is too long. A better approach is to use the built‑in Change Feed, which provides a ordered log of inserts and updates per logical partition.

Thesis

By pairing Cosmos DB Change Feed with the Azure Functions trigger and configuring the lease container correctly, you get exactly‑once processing, automatic load balancing, and a clear checkpointing strategy—while being aware of limits such as delete capture and retention windows.

Section 1: How Change Feed works

Change Feed exposes a per‑partition log of all inserts and updates (deletes are omitted unless you enable the preview "capture deletes" option). Each partition maintains its own cursor, and the Change Feed Processor library (or the Functions trigger) manages leases that store continuation tokens. This allows multiple worker instances to scale out without losing or duplicating events.

Key characteristics

• Latency typically under 100 ms for new changes.
• Guaranteed ordering within a logical partition.
• Retention tied to the container’s TTL, with a hard maximum of 30 days.
• Emits the full document post‑image; schema changes appear as new or missing fields.

Section 2: Choosing a consumption model

Three ways to read the feed exist:

1. Pull model – Use the Change Feed Processor library in your own host (e.g., a VM or Kubernetes pod). You control checkpoint storage and can implement custom logic.
2. Push model (recommended for most scenarios) – Azure Functions Cosmos DB trigger. The runtime creates a lease container, handles load balancing, and invokes your function for each batch of changes.
3. Raw SDK iteration – For edge cases where you need to manage continuation tokens yourself (e.g., integrating with a non‑Azure host).

For event‑driven architectures, the Functions trigger reduces operational overhead while still giving you exactly‑once semantics.

Section 3: Lease container and checkpointing considerations

The lease container stores a document per logical partition containing the continuation token. Throughput on this container should be provisioned separately; a typical starting point is 400‑1000 RU/s. If the lease container is throttled, checkpoint updates lag, causing functions to reprocess the same batch after a lease renewal.

Practical configuration

# Example Azure CLI commands (placeholders in <>)
# 1. Create Cosmos DB account (Core SQL API)
az cosmosdb create \
  --name <accountName> \
  --resource-group <resourceGroup> \
  --locations regionName=<region> \
  --default-consistency-level Session

# 2. Create a container with Change Feed enabled (default)
az cosmosdb sql container create \
  --account-name <accountName> \
  --resource-group <resourceGroup> \
  --database-name <dbName> \
  --name <containerName> \
  --partition-key-path /myPartitionKey \
  --throughput 400

# 3. (Optional) Enable capture deletes – preview feature as of 2024
az cosmosdb sql container update \
  --account-name <accountName> \
  --resource-group <resourceGroup> \
  --database-name <dbName> \
  --name <containerName> \
  --capture-deletes true

# 4. Create Function App (Consumption plan)
az functionapp create \
  --resource-group <resourceGroup> \
  --name <functionAppName> \
  --storage-account <storageAccount> \
  --runtime dotnet \
  --functions-version 4

# 5. Add Cosmos DB trigger (via VS Code or Azure CLI)
# The trigger JSON will reference the lease container; you can set leaseContainerName explicitly.

After deploying the function, inspect the lease container in Data Explorer. You should see documents with ids like Lease<partitionKeyRangeId> containing a continuation property. When you insert a document into the source container, the function logs a receipt and the lease document’s continuation token advances.

Worked example: Azure Function in C#

using System;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;

public static class CosmosChangeFeedFunction
{
    [FunctionName("CosmosChangeFeedFunction")]
    public static void Run(
        [CosmosDBTrigger(
            databaseName: "MyDb",
            containerName: "MyContainer",
            Connection = "CosmosDBConnection",
            LeaseContainerName = "leases",   // explicit lease container name
            CreateLeaseContainerIfNotExists = true)]
        IReadOnlyList<Document> changes,
        ILogger log)
    {
        if (changes == null || changes.Count == 0)
        {
            log.LogInformation("No changes in this batch.");
            return;
        }

        log.LogInformation("Received {Count} changes", changes.Count);
        foreach (var doc in changes)
        {
            // Tolerant deserialization: treat as dynamic or JsonNode
            var id = doc.GetPropertyValue<string>("id");
            log.LogInformation("Processing document {Id}", id);
            // Your business logic here
        }
    }
}

Where to run: Deploy this code to the Function App created above. Required permissions: the function’s managed identity (or the key in the CosmosDBConnection app setting) must have read and write on the source container and read/write on the lease container.

Expected checks

• Function App logs show "Received X changes" after each insert/update.
• Lease container documents update their continuation token.
• No duplicate processing of the same document appears in the logs (verify by checking a unique business‑logic side effect, e.g., a counter increment).

Risks and mitigation

• Lease container throttling – Monitor request units on the lease container via Metrics; increase throughput if you see frequent 429 responses.
• Delete events not captured – If you need deletes, enable the preview capture-deletes flag or implement a soft‑delete pattern (e.g., add a deletedAt field and a TTL).
• Schema evolution – Deserialize to a flexible type (System.Text.Json.JsonNode or Newtonsoft.Json.Linq.JObject) to avoid failures when new fields appear.
• Retention window – Change Feed only retains changes for up to 30 days (or the container’s TTL). For longer replay needs, persist events to a durable store (e.g., Event Hub) within the function.

Limitation and practical verification

The preview “capture deletes” feature is not yet generally available; if you rely on it, test in a non‑production environment and be prepared for breaking changes. To verify that deletes are being captured, insert a document, note its _ts, then delete it and check the function logs for a change feed item where the document retains its id but may have a deleted property (depending on how you model soft deletes).

Actionable closing

Start small: enable Change Feed on a low‑traffic container, deploy a Functions trigger with a dedicated lease container, and observe the lease documents updating as you write data. Once you confirm exactly‑once processing, gradually increase throughput and consider adding soft‑delete handling or event persistence for longer‑term replay. This approach gives you a reliable, low‑latency event stream without the operational burden of custom polling or checkpoint code.