Aerospike Strong Consistency Mode Behaves Differently Across Local and Production Clusters

Answer

The configuration boundary is the per-namespace strong-consistency setting combined with quorum acknowledgment and heartbeat/fabric timeouts. Local LANs absorb brief blips; production links with higher latency can miss heartbeats, trigger partition-ownership changes, and drop strong-consistency availability because a quorum of replicas cannot acknowledge. Production heartbeat timeouts may be increased relative to local values, but the same values must be applied to every node in the cluster. Within one cluster, namespace settings—strong-consistency, replication-factor, rack-id, and rack-aware—must match on all nodes before deployment; rack settings do not need to match across separate clusters.

Confirmed facts

• Strong consistency is configured per namespace and must be identical on every node in the same cluster.
• SC writes and reads require replica acknowledgment, so minority isolation can make the namespace unavailable even when AP mode would continue serving.
• Heartbeat interval and timeout must be symmetric across all nodes; asymmetric failure detection can cause split-brain suspicion and inconsistent partition ownership.
• Replication factor, rack-id, rack-aware, and namespace name should match across nodes in the same cluster.

Likely explanation (unverified)

Production nodes likely span racks or data centers with higher latency than the local three-node LAN. Default heartbeat timeouts that pass locally may expire during production network blips, causing node leave/join events. Each topology change can temporarily reduce available replicas below the SC quorum and produce availability drops. Verify this with logs before changing timeouts.

Steps

1. On every local and production node, run asadm -c "namespace" -r <namespace> -s and diff strong-consistency, replication-factor, rack-id, and rack-aware.
2. Run asinfo -v "heartbeat:interval" and asinfo -v "heartbeat:timeout" on every node; confirm all values match.
3. If production timeouts differ from local values, set identical heartbeat-interval and heartbeat-timeout in aerospike.conf on every production node, then restart Aerospike.
4. Inspect Aerospike logs during a blip for partition-ownership changes, replica write timeouts, and node leave/join events correlated with heartbeat misses.

Missing diagnostic detail: current heartbeat-timeout and replication-factor values on production nodes versus the local cluster. This changes the recommendation only if production already uses longer timeouts, which would point to rack placement or replica count instead of heartbeat tuning.

Assumes Aerospike Enterprise Edition, since strong consistency is an enterprise feature, and that asadm/asinfo are available. Command syntax varies by version; verify against your release before applying changes.