Architecting Couchbase XDCR for Regional Read Scaling and Disaster Recovery

The Challenge: Balancing Availability and Data Locality

Organizations often face a trade-off between maintaining a low Recovery Point Objective (RPO) and providing low-latency reads to users in different geographic regions. Relying on a single cluster creates a single point of failure and forces cross-region network hops for every request. The solution is Cross-Data Center Replication (XDCR), which asynchronously mirrors data between independent Couchbase clusters.

The primary takeaway is that XDCR is asynchronous. This means there is always a window of potential data loss between the source and target, and applications must be designed to handle eventual consistency.

The Minimal Viable Architecture

To achieve a production-ready footprint that supports both disaster recovery (DR) and regional scaling, the smallest suitable design consists of two independent clusters. Each cluster must maintain its own quorum to avoid a total outage if the inter-cluster link fails.

Cluster Configuration

• Node Count: Minimum of 3 nodes per cluster. This ensures that the loss of a single node does not compromise the cluster's ability to maintain a majority for internal operations.
• Bucket Design: One bucket per workload. Each bucket should have replica counts configured based on local availability needs (typically 1 or 2 replicas) to protect against local disk failure.
• Storage: No shared storage between clusters. Each cluster must have its own independent disks to prevent a storage-layer failure from impacting both regions.

Replication Stream Setup

XDCR is configured as a stream from a source bucket to a target bucket. For a DR and read-scaling scenario, a unidirectional stream (Source $\rightarrow$ Target) is the simplest and safest approach. If the target region also needs to accept writes, a bidirectional setup is required, though this introduces significant conflict resolution complexity.

Trust and Data Boundaries

In an XDCR architecture, trust boundaries are cluster-local. Because replication traffic often traverses public or shared corporate networks, the following boundaries must be enforced:

• Administrative Isolation: Use separate administrator credentials for each cluster. A compromise of the source cluster's admin credentials should not grant automatic access to the target cluster.
• Encryption in Transit: All XDCR traffic must be encrypted using TLS. This prevents eavesdropping on data as it moves between geographic regions.
• Network Segmentation: Limit the target cluster's firewall to only accept XDCR traffic from the specific IP addresses of the source cluster's nodes.

Operational Checks and Health Monitoring

Because XDCR happens in the background, a failure in replication may not be immediately apparent to the application. Monitoring must focus on the "lag" between regions.

Key Metrics to Track

Metric	Significance	Actionable Threshold
Replication Lag	Time difference between source mutation and target application.	Exceeding the defined RPO (e.g., > 5 minutes).
Backlog Size	Number of mutations waiting to be sent.	Sustained growth indicates network saturation or target bottleneck.
Mutation Errors	Count of documents that failed to replicate.	Any non-zero increase requires log inspection for data corruption.

Verification Command

To verify the health of a replication stream, use the Couchbase REST API from a management server with administrative permissions. Replace <cluster-ip> with the target cluster's IP and <bucket-name> with your specific bucket.

curl -u admin:password http://<cluster-ip>:8091/pools/default/buckets/<bucket-name>/xdcrReplication

Expected Result: The response should show the replication status as active and the lag value should be within your acceptable RPO window.

Failure Modes and Design Shifts

The current design assumes a mostly stable network and a primary-secondary write pattern. Certain conditions require a change in architecture:

1. Network Partition

If the link between regions breaks, XDCR will stall. The source cluster will continue to accept writes, increasing the backlog. Once the link is restored, the target cluster may experience a CPU and I/O spike as it processes the accumulated backlog. If the backlog is too large to clear within the RTO, a full bucket restore from backup may be necessary.

2. Source Cluster Total Loss

In a total outage of the source region, any data mutated but not yet replicated (the lag window) is lost. To mitigate this, you must move the application's write traffic to the target cluster. This is a manual or orchestrated failover; automated failover is risky as it can lead to "split-brain" scenarios where both clusters accept writes for the same document.

3. Bidirectional Conflict Resolution

If the design shifts to Active-Active (writes in both regions), you must define a conflict resolution policy. Couchbase typically uses Last Write Wins (LWW) based on a hybrid logical clock. If your application requires business-logic-based resolution (e.g., merging two shopping carts), you must implement a custom resolution layer at the application level, as the database will only preserve one version of the document.

Rollback and Recovery

Because XDCR does not change the state of the source data, "rolling back" a replication stream simply involves deleting the XDCR definition on the source cluster. However, this does not remove the replicated data from the target cluster. To completely revert the target state, you must manually purge the target bucket or delete and recreate it.