Can a custom indexing policy reduce request-unit costs while preserving range-query performance in Cosmos DB?
0 reputation · 05 May 2025, 18:59 UTC
0 reputation · 05 May 2025, 18:59 UTC
Automatic indexing in Azure Cosmos DB for NoSQL indexes every property on write, simplifying schema evolution but consuming request units per operation. When a custom indexing policy excludes specific paths, the default composite index interaction with remaining fields becomes a design consideration, particularly for range and equality queries. The policy change triggers an index transformation that may introduce temporary latency proportional to item count and write throughput.
This raises the question of whether query functionality for commonly accessed fields can be maintained without incurring proportional RU overhead or availability risk. Does disabling automatic indexing on underutilized paths preserve equality-filter query performance? Can the index transformation be performed incrementally to avoid throttling? What observable metrics signal that the new indexing policy has fully taken effect?
30275 reputation · 06 May 2025, 02:49 UTC
Yes. A custom indexing policy that excludes genuinely unused paths reduces write request-unit (RU) charges, because Cosmos DB no longer maintains index entries for those paths on every insert or update. Range-query performance is preserved as long as every path you filter or sort on with >, <, >=, <=, or ORDER BY keeps a Range index. The risk is not the policy itself but accidentally excluding a path a query depends on, and underestimating the background index transformation that follows the change.
With the default automatic policy, Cosmos DB indexes every scalar property, and each indexed property adds to the per-write RU charge. Switching the indexing mode to consistent with an explicit excludedPaths list (typically /unusedSection/* style wildcards) removes that per-write cost for the excluded subtrees. Equality filters (=, IN, ARRAY_CONTAINS) work with either Hash or Range indexes, so keeping a Range index on your hot equality fields preserves their performance and future-proofs them for range use.
The important caveat: composite indexes do not substitute for single-path range indexes. The default policy's wildcard composite entry supports ORDER BY patterns, but a simple range predicate on one field needs a Range index on that field. If you exclude a path that a range query uses, the query still returns correct results but falls back to a scan — RU consumption and latency climb sharply.
Applying a new policy triggers an asynchronous background rebuild. Writes are not blocked, but queries that depend on newly added indexes may return incomplete results until the transformation finishes. There is no incremental or phased API for this; your only levers are:
Also note: re-including a path you excluded earlier triggers another full transformation with the same cost, so exclude conservatively.
ReadContainerAsync in the SDK, or REST GET on the collection) and watch indexTransformationProgress until it reaches 100%. This is not an Azure Monitor metric, so alerting requires custom polling.retrievedDocumentCount close to outputDocumentCount. A large retrieved-to-output ratio means you are scanning.x-ms-request-charge header on create/replace operations before and after; the reduction should track the number of excluded indexed paths.Exclude only paths you are confident no query filters, sorts, or joins on — audit your query workload first using diagnostic logs or SDK query text capture. Keep Range indexes on everything queried, add composite indexes only for multi-field ORDER BY or multi-predicate queries, and treat the transformation as a maintenance event with monitoring, not a fire-and-forget change. Behavior described reflects the NoSQL API's long-standing indexing model; if you are on a recently released SDK or serverless tier, re-verify transformation progress fields against current documentation before relying on them in automation.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.