Designing a Globally Distributed Low‑Latency Store with Azure Cosmos DB Multi‑Region Writes

Problem and Takeaway

Applications that serve users across continents often need sub‑second read and write latency while tolerating eventual consistency. Azure Cosmos DB’s multi‑region write capability with automatic conflict resolution satisfies this, but only when the account, container, and operational guardrails are configured deliberately. The smallest viable design is a single Cosmos DB account with two or more write‑enabled regions, a well‑chosen partition key, autoscale (or serverless) throughput, and the default Last Writer Wins (LWW) policy—no custom code required unless business rules demand it.

Requirements Checklist

• Latency target: < 100 ms read/write per region.
• Consistency model: Eventual or bounded staleness acceptable; strong consistency would negate multi‑region write benefits.
• Conflict strategy: Deterministic resolution (LWW) or a version‑controlled custom stored procedure.
• Data‑residency/compliance: Certain regions must be excluded for regulated data (GDPR, HIPAA, etc.).
• Cost envelope: Cross‑region replication adds RU consumption; validate with the Azure pricing calculator before production.

Smallest Suitable Design

Account & Regions

Create one Cosmos DB account (Core (SQL) API) and enable enableMultipleWriteLocations. Add at least two regions with write capability, e.g., East US and West Europe. Each region becomes a trust boundary; data written in one region replicates asynchronously to the others.

Container & Partition Key

Choose a partition key that distributes load evenly across logical partitions—typically a high‑cardinality field such as userId or deviceId. Provision the container with autoscale max RU/s (e.g., 4000 RU/s max) or serverless if traffic is sporadic.

Conflict Resolution Policy

Default LWW uses the _ts system timestamp. If business logic requires “last update wins per entity” but with a custom tie‑breaker, implement a stored procedure and reference it in the container’s conflictResolutionPolicy. Keep the procedure version‑controlled; any silent bug can cause data loss.

Example: Azure CLI Provisioning

# Run in Azure Cloud Shell or a terminal with Azure CLI ≥ 2.45, logged in as Owner or Contributor on the subscription.
# Replace placeholders with your values.
RG=rg-cosmos-global
ACCOUNT=cosmos-global-app
LOCATIONS=("East US" "West Europe")
DB=appdb
CONTAINER=orders
PARTITION_KEY=/userId

az group create -n $RG -l eastus

az cosmosdb create \
  -g $RG -n $ACCOUNT \
  --default-consistency-level Eventual \
  --enable-multiple-write-locations true \
  --locations regionName="East US" failoverPriority=0 isZoneRedundant=false \
  --locations regionName="West Europe" failoverPriority=1 isZoneRedundant=false

az cosmosdb sql database create -g $RG -a $ACCOUNT -n $DB

az cosmosdb sql container create \
  -g $RG -a $ACCOUNT -d $DB -n $CONTAINER \
  --partition-key-path $PARTITION_KEY \
  --throughput 4000 \
  --conflict-resolution-policy mode=LastWriterWins

Permissions: The executing identity needs Microsoft.DocumentDB/databaseAccounts/write and Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write on the resource group. Risk: Enabling multi‑region writes immediately doubles write RU consumption for each replicated write; monitor cost during the first 24 h.

Trust / Data Boundaries

Each write‑enabled region is a trust boundary. To enforce residency:

• Apply an Azure Policy that denies creation of Cosmos DB write locations in non‑approved regions.
• Use resource locks (CanNotDelete) on the account to prevent accidental region addition.
• Tag the account with compliance=gdpr and audit via Azure Security Center.

If a region must be read‑only for compliance, set its failoverPriority but do not enable write capability; the portal UI shows a “Read‑only” badge for that location.

Operational Checks

Metric / Check	Tool	Threshold / Action
Replication Latency	Azure Monitor → Metrics → “Replication Latency”	> 500 ms sustained → investigate network or throttle.
Conflicts Count	Metrics → “Conflicts” + Conflict Feed (change feed)	> 0 per minute for LWW → review custom resolver logic.
RU/s Utilization	Metrics → “Normalized RU Consumption”	> 80 % of autoscale max → consider raising max or redesign partition key.
Region Failover	Activity Log / Azure Service Health	Automatic failover triggered → verify write availability in remaining regions.

Configure alert rules on the above metrics (e.g., replication latency > 1 s for 5 min) to receive email or webhook notifications.

Failure Modes & Mitigations

1. Region outage: Reads automatically route to the next healthy region; writes continue in remaining write‑enabled regions. No manual failover needed.
2. Network partition: Temporary latency spikes and stale reads under eventual consistency. Mitigate by setting a bounded staleness level (e.g., 5 s) if the application can tolerate it.
3. Conflict resolution failure: With a custom stored procedure, an unhandled exception routes conflicting documents to the conflict feed. Monitor the feed and have a runbook to inspect and re‑apply.
4. Throughput exhaustion: Autoscale hits max RU/s; requests return 429. Enable retry logic with exponential backoff in the SDK (default) and consider raising the autoscale ceiling.

Conditions That Would Change the Design

• Strong consistency required: Switch to a single write region or use the “Strong” consistency level; this disables multi‑region writes.
• Regulatory change: New data‑residency law forces removal of a write region—disable write capability via az cosmosdb update --locations and re‑balance traffic.
• Conflict semantics evolve: Business decides “merge” instead of LWW. Implement a custom resolver, add integration tests, and deploy via CI/CD; expect added operational overhead.
• Cost overrun: Replication RU consumption exceeds budget. Options: reduce write regions, lower autoscale max, or move to a single write region with read replicas.

Verification Steps (Run After Deployment)

# 1. Confirm multi‑region writes enabled
az cosmosdb show -g $RG -n $ACCOUNT --query "{multiWrite:enableMultipleWriteLocations, locations:locations[].{name:name, write:failoverPriority}}" -o table

# 2. Inspect container conflict policy
az cosmosdb sql container show -g $RG -a $ACCOUNT -d $DB -n $CONTAINER \
  --query "conflictResolutionPolicy" -o json

# 3. Check replication latency (last 30 min)
az monitor metrics list \
  --resource $ACCOUNT \
  --resource-group $RG \
  --resource-type Microsoft.DocumentDB/databaseAccounts \
  --metric "ReplicationLatency" \
  --interval PT30M --aggregation Average

# 4. Verify conflict feed is empty (LWW)
az cosmosdb sql container show -g $RG -a $ACCOUNT -d $DB -n $CONTAINER \
  --query "conflictResolutionPolicy.mode" -o tsv
# Expect "LastWriterWins"

Expected checks: The first command should show multiWrite: true and each location with a numeric failoverPriority. The conflict policy must read LastWriterWins unless a custom procedure was added. Replication latency should stay under your SLA threshold (e.g., 200 ms).

Limitations & Practical Validation

• Changing the consistency level after creation may require a full data migration; decide early.
• Custom conflict resolvers add code deployment complexity and must be version‑controlled; a bug can silently drop writes.
• Cross‑region replication cost is not trivial—run the Azure pricing calculator with your projected write volume and region count before committing.
• Zone‑redundant writes (isZoneRedundant=true) increase availability within a region but also raise RU cost; evaluate per‑region SLA needs.

Practical validation: after a week of production traffic, export the ReplicationLatency and Conflicts metrics to a Log Analytics workspace and run a Kusto query to confirm 99th‑percentile latency < 300 ms and zero conflicts for LWW. If either metric deviates, revisit partition key distribution or region selection.

Diagram Labels

• Cosmos DB Account (multi‑region writes enabled)
• Write‑enabled Regions (East US, West Europe)
• Autoscale Container with Partition Key
• Conflict Resolution Policy (LWW or Custom)