Diagnosing Aerospike XDR Lag and Data Inconsistency in Multi‑Site Deployments

Recognizable Condition

In a multi‑site Aerospike deployment (Aerospike Database 6.x‑7.x) you observe one or more of the following:

• The XDR lag metric (xdr_lag) for a destination datacenter (DC) exceeds the configured threshold (e.g., >5000 ms).
• Client applications on the source cluster report increased timeout errors.
• The xdr_error_counts statistic rises, or asadm shows XDR_WARN/XDR_ERROR lines in aerospike.log.
• Destination reads return stale data compared with the source.

These symptoms indicate that Cross‑Datacenter Replication (XDR) is falling behind or encountering errors that prevent timely shipping of records.

Cause/Diagnostic Table

Possible Cause	Observable Symptom	Where to Look
Network partition or latency spike between DCs	Increased xdr_ship_latency, TCP retransmits on the link	Source: asinfo -v 'statistics'; OS: ss -ti or sar -n DEV
Destination namespace write saturation	Destination write_q or proxy_write_q high, rising xdr_remote_node_write_errors	Destination: asinfo -v 'namespace/'
Source namespace partition skew (hot partitions)	Single XDR thread per partition overwhelmed, lag concentrated on specific partitions	Source: asadm -e 'show xdr' (per‑node thread status)
XDR filter/transform misconfiguration	High xdr_filtered_records, ship attempts dropped	Source: asinfo -v 'xdr:dc=*' and review filter expression
TLS/certificate expiry on destination	Spike in xdr_tls_handshake_failures in logs	Source: aerospike.log; destination TLS cert validity

Ordered Diagnostic Checks

1. Confirm lag per DC and namespace on the source cluster (requires cluster‑admin read access):

   asinfo -v 'xdr:dc=*'

   Look for any DC where xdr_lag exceeds your threshold.

2. Examine XDR‑specific statistics:

   asinfo -v 'statistics'

   Focus on xdr_ship_latency, xdr_throughput, and xdr_error_counts.

3. Check the destination namespace for queue pressure:

   asinfo -v 'namespace/'

   Replace <ns> with the namespace name. Note write_q and proxy_write_q percentages.

4. Show per‑node XDR thread state to spot stalled threads:

   asadm -e 'show xdr'

   Each line includes STATE (SHIPPING, SLEEPING, STALLED) and error counters.

5. Correlate with OS‑level network metrics (run on the source host):

   ss -ti | grep -E 'retrans|rto'

   sar -n DEV 1 5

   Look for increasing retransmits or interface drops coinciding with XDR lag spikes.

6. Scan the Aerospike log for XDR‑related warnings/errors (requires file read permission):

   grep -i 'XDR_WARN\|XDR_ERROR' /var/log/aerospike/aerospike.log | head -20

   Timestamps help align with metric spikes.

Fixes Tied to Findings

• Network issues: Increase the number of concurrent XDR connections to better utilize the link.

  asinfo -v 'set-config:context=xdr;xdr_max_concurrent_connections=256'

  (Adjust based on ulimit -n; raising beyond 1024 may exhaust file descriptors.) Enable TCP BBR or tune keepalive (tcp_keepalive_time, tcp_keepalive_intvl) on the OS.
• Destination write saturation: Add write threads or enlarge the write queue on the destination node.

  asinfo -v 'set-config:context=namespace;id=;write-threads=8'

  asinfo -v 'set-config:context=namespace;id=;write-queue-size=128000'

  If queues remain high, consider adding nodes to the destination cluster.
• Partition skew: Enable per‑partition parallelism (Aerospike 7.0+).

  asinfo -v 'set-config:context=xdr;xdr-partition-parallelism=true'

  Ensure the destination cluster is also 7.0+; mixed versions cause protocol errors. Alternatively, rebalance the source namespace:

  asinfo -v 'rebalance'

• Filter/transform misconfiguration: Audit the XDR filter expression.

  asinfo -v 'xdr:dc=:filter-expression'

  If the expression uses a Lua UDF that is costly or erroneous, simplify or remove it, then reload:

  asinfo -v 'set-config:context=xdr;id=:filter-expression=''

• TLS/certificate expiry: Refresh the XDR TLS context without a full restart (Aerospike 6.2+).

  asinfo -v 'xdr:dc=:tls-refresh'

  For older versions, perform a rolling node restart after placing new certs in /etc/aerospike/.

Escalation Criteria

Escalate to senior engineering or vendor support when any of the following persist after initial fixes:

• XDR lag exceeds 2 × xdr-ship-interval for >15 minutes.
• xdr_error_counts > 1 % of shipped records over a 5‑minute window.
• Destination write_q > 80 % for >5 minutes despite thread/queue increases.
• Digestlog comparison shows mismatched digests for the same partition/set (see verification).
• Log contains XDR_THREAD_STALLED entries that do not clear after thread‑state checks.

Verification After Fix

1. Monitor lag every 30 seconds for 10 minutes:

   watch -n 30 'asinfo -v "xdr:dc=*"'

   Confirm xdr_lag trends toward zero and stays below your threshold.
2. Check XDR thread health:

   asadm -e 'show xdr' | grep -E 'STATE=SHIPPING'

   All threads should show STATE=SHIPPING with zero error counters for five consecutive polls.
3. Validate destination queue health under normal load:

   asinfo -v 'namespace/'

   Ensure write_q and proxy_write_q remain < 50 %.
4. Perform a limited digestlog spot‑check (run during low‑traffic windows):

   asinfo -v "xdr:dc=:digestlog:namespace=:set=:partition=
   "

   Execute on source and destination; the returned digests must match.
5. Confirm absence of new XDR warnings/errors:

   grep -i 'XDR_WARN\|XDR_ERROR' /var/log/aerospike/aerospike.log | wc -l

   Count should be zero for a 30‑minute observation window.

Limitations and Practical Checks

• Digestlog comparison is I/O‑intensive; limit it to suspect partitions and schedule during off‑peak hours.
• Increasing xdr_max_concurrent_connections beyond 1024 per node may exhaust file descriptors; verify ulimit -n is sufficient (e.g., 65535) before raising the value.
• xdr-partition-parallelism requires Aerospike 7.0+ on both source and destination; enabling it on a mixed‑version cluster will cause protocol errors visible as XDR_PROTOCOL_ERROR in logs.
• TLS refresh via xdr:tls-refresh is only available from Aerospike 6.2; older versions need a rolling restart, which incurs brief service interruption.

By following the ordered checks, applying the targeted fix, and verifying with the steps above, you can isolate the root cause of XDR lag, restore timely replication, and ensure data consistency across your datacenters.