Indexing policy revert and query consistency during partial re-indexing

Question

When an indexing policy is modified on a Cosmos DB container, a background re‑indexing job begins while the container continues to accept reads and writes. The exact moment when newly indexed properties transition from partially indexed to fully queryable is not exposed as a single observable flag, and query results may return items indexed before the policy change alongside items that have not yet been re‑indexed. This creates a result‑set inconsistency window where predicates on the new attribute can yield different outcomes depending on each item's modification timestamp relative to the policy shift. Additionally, the Change feed emits per‑partition completion events, but it remains unclear whether these events guarantee that all items within a partition have reached a query‑consistent state or if in‑flight writes during re‑indexing alter the boundary. Given that abruptly stopping the job can leave items in an inconsistent index state, the operational path for a safe policy revert is ambiguous.

Under what conditions does a query on a newly indexed property guarantee result consistency across all container items? Can the Change feed alone be relied upon to signal a safe point for policy revert, or is additional tracking required? What is the impact on ongoing writes during re‑indexing regarding the visibility of index updates?

Tasadduq Burney · Answer

Answer

Assumption: Azure Cosmos DB SQL API container with consistent indexing mode and online re-indexing; behavior can differ by API and tier. A query on a newly indexed property is result-consistent across all items only after index transformation has reached 100 percent across every partition and the container is using consistent indexing mode. Until that point, read consistency levels such as session or strong do not fill index gaps; they only govern read propagation. The Change feed cannot by itself mark a safe revert point because it records item changes, not index backfill completion, and items not modified since the policy change may never appear in the feed. Ongoing writes remain accepted and are indexed under the current policy in consistent mode, but they compete for request units and can increase latency or throttling; in-flight writes can straddle the old and new policy boundary.

Confirmed behavior

Index transformation progress, not Change feed checkpoints, is the governing signal for query coverage of a new property.
Strong or session consistency does not make a partially built index complete.
Lazy indexing mode does not automatically index new or updated items; a manual indexing request is required.
Record the policy-change time as a write watermark; Change feed can reveal writes after that watermark but cannot prove backfill of pre-existing items.
Reverting while transformation is in progress can leave the new property partially indexed and make predicates unreliable.

Likely explanation for the inconsistency window

Items modified before the policy change are indexed under the old policy until background re-indexing reaches them. Items modified after the change are indexed under the new policy. During the interval, a predicate on the new property can match items in some partitions and miss equivalent items in others, even if the read consistency level is strong.

Steps for this case

Confirm the container is in consistent indexing mode; if it is lazy, stop and plan manual indexing before relying on the new property.
Monitor index transformation progress in the Azure portal or SDK diagnostics until it reports 100 percent across all partitions.
Validate the new predicate with a count or sample query against an independent source such as a full partition scan or backfilled metadata store.
Only after completion and validation, revert the indexing policy. Keep any dual-write or feature-flag code in place until both old and new properties are no longer required.

Verification

Check the container's index transformation percentage and confirm 100 percent across all partitions. Inspect Cosmos DB metrics or SDK diagnostics for indexing progress, indexing latency, and rate-limited requests during and after the change. Confirm which metric exposes index transformation progress for your API and tier before automating the revert.

Missing diagnostic

Confirm the indexing mode and whether index-transformation progress is exposed for your Cosmos DB API and account tier; this determines whether waiting for 100 percent is sufficient or whether manual indexing is required. Treat this as a draft for human review until that diagnostic is confirmed.