XDCR Replica Write Policy Transition in Aerospike 7.3: Unresolved Consistency Trade‑offs

Direct Answer

Enabling use-async-write (or setting xdcr-replica-write-policy to async) in Aerospike 7.3 decouples remote replica completion from source acknowledgment. You can claim only eventual cross-datacenter consistency. Primary-ack timestamps in your logs do not prove replica ordering or durability.

Confirmed Behavior vs. Likely Failure Cause

Confirmed

• Async XDCR shipping means the source returns to the client before the remote apply finishes.
• Cross-cluster per-record ordering is not guaranteed in async mode; a later write can be applied while an earlier write is delayed or retried.
• Namespace type changes the local durability/loss window, not the async XDCR ordering contract.
• Memory-only namespaces add a larger acknowledged-write loss window after restart or failover; persistent namespaces can recover local state, but remote replicas can still be stale.
• Local replication-factor or strong-consistency settings do not extend linearizability across XDCR.

Likely Explanation for xdcr: replica write failed

The most probable causes are remote apply failures: namespace or set mismatch, authentication errors, record size/version conflicts, destination throttling, or an unavailable destination cluster. Correlate these entries with destination logs and XDCR lag metrics rather than treating source timestamps as ordering evidence.

Consistency and Ordering by Namespace Type

Async XDCR is eventual across clusters for all namespace types. The difference between memory-only and persistent namespaces is the size of the local loss window after a restart, not the cross-datacenter consistency level. Neither configuration makes async XDCR linearizable or read-your-writes across clusters. Strong-consistency namespaces do not automatically extend that guarantee across XDCR.

Retry and Backpressure Guidance

There is no defensible universal retry-limit threshold. Size retries and backoff for transient faults, and enforce a lag SLO with backpressure. For sustained replica failures, throttle or pause shipping rather than raising retries indefinitely; higher retry limits can increase latency, queue growth, and throttling.

Verification Steps

1. Confirm whether the affected namespaces are AP memory-only, AP persistent, or strong-consistency. This is the missing diagnostic detail that changes the recommendation.
2. Check the running configuration for the exact XDCR and namespace settings on your build, including async policy and retry-related parameters, then confirm against the 7.3 release notes.
3. Run a per-key ordering test: write k=v1 then k=v2, pause the destination, resume, and verify whether v2 can appear before v1 or be overwritten by v1.
4. Measure throttling by observing source write latency, XDCR queue depth/lag, and retry/failure counters while injecting destination failures.

Treat 7.3 parameter names, defaults, and exact semantics as version-sensitive; verify them against your exact server build before changing production configuration.