Configuring Aerospike Multi‑Topology Replication for Cross‑Data‑Center Consistency

Desired Outcome

Set up a robust, cross‑data‑center Aerospike cluster that automatically replicates records from a primary site to one or more remote sites, maintaining consistency and providing failover without manual re‑balancing.

Prerequisites

• Two or more Aerospike clusters (primary + remote) running the same major version.
• Each node has a static IP address reachable from all other sites.
• Firewall allows traffic on port 3000 (client) and 3001 (replication).
• Network latency between sites measured < 100 ms for synchronous replication; higher latency requires adjusting replication‑timeout.
• Same replication-factor value on every node in each cluster.

Step 1 – Define the Topology File

The topology file lists every remote cluster and its replication factor. Place the file on each node in the primary cluster and copy it to the remote clusters.

# /etc/aerospike/topology.conf
# Format:     ...
# Example: remote1 2 10.0.1.10 10.0.1.11

remote1 2 10.0.1.10 10.0.1.11
remote2 2 10.0.2.20 10.0.2.21

Replace the IP addresses and cluster names with your environment. The file must be readable by the Aerospike daemon (typically owned by root:aerospike with 640 permissions).

Step 2 – Configure Each Aerospike Node

Modify /etc/aerospike/aerospike.conf on every node to point to the topology file and set the replication factor. Example snippet for a node in the primary cluster:

cluster {
    name primary
    replication-factor 2
    topology /etc/aerospike/topology.conf
}

For nodes in a remote cluster, use the same replication-factor value and the same topology file. The topology directive is optional on remote nodes if they share the same file path; otherwise, copy the file to the same location.

Permissions and Restart
After editing, run:

sudo systemctl restart aerospike

Verify the daemon started with systemctl status aerospike and check the log for any topology parsing errors.

Step 3 – Verify Replication Status

On a node in the primary cluster, run:

asinfo -v replication

Expected output contains entries like:

replication-status: OK
replication-lag: 0

To confirm that records are visible on the remote site, perform a simple write and read:

# Write a test record on the primary cluster
asadm -e "write test_ns test_set key1 string 'value1'"

# Read from a node in the remote cluster
asadm -e "select test_ns test_set key1"

The remote read should return the written value after the replication interval (default 1 s). Use asinfo -v status to check overall health and look for replication-status: OK.

Step 4 – Monitor Latency and Lag

High inter‑site latency can trigger replication timeouts. Monitor with:

asinfo -v replication-lag

Use Aerospike’s asinfo -v topology to confirm every node lists the correct remote clusters. If a node reports replication-lag > 0 or replication-status: FAILED, check:

• Firewall or NAT blocking port 3001.
• Network path quality; run ping and traceroute between sites.
• Consistent replication-factor across all nodes.

Step 5 – Failover and Recovery

When the primary site becomes unavailable, clients should be directed to a remote node. Update the client’s policy to point to the new primary node’s IP. Aerospike’s client libraries automatically discover the cluster topology, but the application may need to re‑configure the seed list.

After failover, re‑balance the cluster if necessary. Aerospike automatically re‑routes record placement based on the topology; manual rebalancing is rarely required. If you need to force a rebalance, use:

asadm -e "rebalance"

Monitor the rebalance progress with asinfo -v status and ensure replication-status: OK returns.

Recovery Options

• Topology Correction: If a node reports an incorrect topology, replace the topology file and restart Aerospike.
• Replication Timeout: Increase replication-timeout in aerospike.conf if latency spikes, but monitor for increased lag.
• Rollback: If a misconfiguration causes data loss, restore the last known good topology file and configuration from backup.

Limitations and Practical Checks

• Multi‑Topology Replication is only supported on Aerospike Enterprise editions. Verify your license before enabling.
• Replication factor must be identical across all nodes; mismatches lead to split‑brain and data loss.
• Always test in a staging environment before production deployment.
• Use asinfo -v replication and asinfo -v topology to periodically validate configuration.

By following these steps, you can confidently deploy cross‑data‑center Aerospike clusters that automatically replicate data, maintain consistency, and provide graceful failover.