Handling Real-Time Data Sync with Cosmos DB Change Feed

The Polling Problem

Many teams keep secondary systems like Elasticsearch or a reporting database in sync with Cosmos DB by polling for updated timestamps. Polling too often wastes Request Units and raises cost. Polling too slowly adds latency to the user experience.

The Cosmos DB Change Feed provides a persistent, ordered log of changes. Instead of asking the database what changed, an application listens for a push of changes as they happen. The Change Feed turns the database from a passive store into an event source.

How the Change Feed Operates

The Change Feed reflects the container transaction log. Inserts and updates are appended to the log. Processing at scale uses a Lease Container.

The lease container stores state about which parts of the change log have been processed by which worker. This allows multiple compute instances to share work without duplicate processing or missed events. If a worker fails, another worker can take the lease and continue from the last checkpoint.

Worked Example: Syncing Orders to a Search Index

A common use case is updating a search index when an order is created or modified. An Azure Function with a Cosmos DB trigger handles this with managed lease scaling.

Configuration Requirements

• Source Container: orders
• Lease Container: orders-leases
• Trigger: Cosmos DB Trigger for SQL API

{
  "bindings": [
    {
      "type": "cosmosDBTrigger",
      "name": "input",
      "direction": "in",
      "leaseCollectionName": "orders-leases",
      "connectionStringSetting": "CosmosDBConnectionString",
      "databaseName": "StoreDatabase",
      "containerName": "orders",
      "createLeaseCollectionIfNotExists": true
    }
  ]
}

Execution Logic

public static void Run([CosmosDBTrigger(
    databaseName: "StoreDatabase",
    containerName: "orders",
    Connection = "CosmosDBConnectionString",
    LeaseCollectionName = "orders-leases")] IReadOnlyList<Order> input, ILogger log)
{
    foreach (var order in input)
    {
        log.LogInformation($"Updating search index for Order ID: {order.Id}");
    }
}

Critical Limitations and Trade-offs

The Deletion Gap

The Change Feed captures inserts and updates only. Deletions generate no event. A soft-delete pattern with an isDeleted flag lets the feed surface the change for downstream removal.

Lease Throughput and Lag

The lease container needs sufficient RU/s. Under-provisioning causes lease steals, pauses, and duplicate delivery. Order is guaranteed per partition key, not globally.

Verification and Monitoring

Monitor Change Feed lag and lease stability in the Azure portal. Growing lag means the consumer cannot keep up with write volume. Frequent lease updates indicate churn and worker instability.

Test by inserting a document into orders and confirming the function logs receive the change promptly.