Implementing Strong Consistency in Aerospike to Prevent Stale Reads

The Problem: Data Divergence During Network Partitions

In standard Aerospike configurations (AP mode), the system prioritizes availability. During a network partition—where nodes cannot communicate but remain online—different nodes may accept writes for the same record. This results in "split-brain" scenarios where clients receive stale data or conflicting versions of a record once the partition heals.

For critical financial transactions, inventory management, or session state, this behavior is unacceptable. You need Linearizability: the guarantee that once a write is acknowledged, every subsequent read will return that value or a more recent one, regardless of which node handles the request.

Prerequisites for Strong Consistency (SC)

Strong Consistency is not a per-record setting; it is a namespace-level configuration. Before enabling SC, ensure your cluster meets these requirements:

• Replica Factor: You must use a replication factor of at least 3. A factor of 1 or 2 cannot maintain the required quorum during a node failure while guaranteeing consistency.
• Version: Aerospike Server 4.0 or higher (verify your specific distribution supports the strong-consistency namespace parameter).
• Network Stability: Because SC relies on synchronous commits, high network jitter will directly increase your application's write latency.

Configuration Procedure

To enable Strong Consistency, you must modify the namespace configuration. Because this changes the fundamental way the cluster handles data replication and commit protocols, it requires a rolling restart of the cluster.

Step 1: Update the Namespace Configuration

Locate your aerospike.conf file on all nodes. Within the namespace block, add or update the strong-consistency parameter.

namespace my_critical_data {
    replication-factor 3
    strong-consistency true
    # Other existing configurations...
}

Step 2: Apply Changes via Rolling Restart

To avoid total cluster downtime, restart nodes one by one. Use a system manager or manual shell commands with root permissions:

# Run on each node sequentially
sudo systemctl restart aerospike

Risk: If you restart too many nodes simultaneously, the cluster will lose quorum and stop accepting writes to the SC namespace.

Operational Comparison: AP vs. SC Mode

The following table outlines the engineering trade-offs when moving from the default Available/Partition-tolerant (AP) mode to Strong Consistency (SC) mode.

Metric	AP Mode (Default)	SC Mode
Write Latency	Low (Asynchronous replication)	Higher (Synchronous quorum commit)
Read Guarantee	Eventual Consistency	Linearizability (No stale reads)
Partition Behavior	Allows writes on both sides	Blocks writes if quorum is lost
Min. Replicas	1	3 (Recommended for availability)

Verification and Diagnostics

After the restart, you must verify that the namespace is operating in SC mode and quantify the latency impact.

Checking Namespace Status

Run the following command using asadm (Aerospike Admin Tool) on any cluster node to confirm the setting:

asadm -e "show namespaces"

Look for the strong-consistency column in the output; it should be marked as true for your target namespace.

Testing for Stale Reads

To verify that the system blocks stale reads during a partition, you can simulate a network failure using iptables on a single node to block traffic to the other cluster members:

# Run on one node to isolate it from the cluster
sudo iptables -A INPUT -p tcp --dport 3000 -j DROP
sudo iptables -A OUTPUT -p tcp --sport 3000 -j DROP

Attempt to read a record from the isolated node. In SC mode, the node will realize it cannot reach a quorum and will return an error rather than serving potentially stale data. To restore connectivity, run sudo iptables -F.

Rollback Procedure

If the increased write latency causes application timeouts, you can revert to AP mode. Warning: Reverting from SC to AP is a state-changing operation that requires another rolling restart.

1. Edit aerospike.conf and set strong-consistency false.
2. Perform a rolling restart of all nodes.
3. Verify the change using asadm -e "show namespaces".