Implementing Strong Consistency in Aerospike for Linearizable Data

The Problem: Eventual Consistency vs. Data Integrity

In distributed databases, the default behavior is often eventual consistency, where a write to one node may take time to propagate to replicas. For applications handling financial transactions, inventory counts, or session locks, this delay can lead to "stale reads," where a client reads an old value immediately after a successful update. To prevent this, Aerospike provides a Strong Consistency mode that guarantees linearizability: once a write is acknowledged, every subsequent read is guaranteed to see that value or a more recent one.

How Strong Consistency Works

Strong consistency shifts Aerospike from a primary-replica model to a quorum-based model. In standard mode, a write is acknowledged as soon as the master node records it. In strong consistency mode, the system requires a quorum—a majority of replicas—to acknowledge the write before it is considered successful. Similarly, reads must contact a quorum of nodes to ensure the data hasn't been updated elsewhere during a network partition.

Configuration Example

Strong consistency is configured at the namespace level in the aerospike.conf file. To enable this, you must set the strong-consistency flag to true. It is critical that your replication factor is set to at least 2 (preferably 3) to allow for a majority quorum.

# Example aerospike.conf snippet
namespace test {
    replication-factor 2
    strong-consistency true
    # consistency-level defines the quorum requirement
    # MAJORITY is the standard for linearizability
    consistency-level MAJORITY
    
    # Other standard namespace settings
    storage-engine device
    memory-size 4G
}

Deployment Note: After modifying the configuration, restart the Aerospike service on all nodes in the cluster to apply the changes. These settings cannot be changed dynamically without a restart.

Verifying the Configuration

To ensure the namespace is operating in strong consistency mode, run the asinfo command from the command line of any node in the cluster. You will need root or sudo permissions to execute this tool.

# Run on any cluster node
asinfo -v "namespace;test;strong-consistency"

Expected Result: The command should return strong-consistency: true. If it returns false, the cluster is still operating in eventual consistency mode.

Performance Trade-offs and Limitations

Implementing strong consistency is not a "free" upgrade; it introduces specific engineering constraints:

• Increased Latency: Because the system must wait for multiple nodes to acknowledge a write or read, the round-trip time (RTT) increases compared to the single-node acknowledgment of eventual consistency.
• Reduced Availability: During a network partition, if a majority of replicas for a specific partition are unreachable, writes to that partition will fail. In eventual consistency, the remaining reachable node could still accept writes.
• Bin Type Restrictions: Strong consistency applies to standard record operations. Be aware that certain complex data structures or User Defined Functions (UDFs) may not maintain these guarantees and could fall back to eventual consistency patterns.

Common Engineering Mistakes

Insufficient Replication Factor

A common failure occurs when strong-consistency is enabled on a namespace with a replication-factor of 1. Since a majority of 1 is 1, but there are no other replicas to provide a quorum during failures, the system becomes extremely fragile. If the single node holding the data is unavailable, there is no failover mechanism, and the strict quorum requirements can lead to unexpected client timeouts.

Ignoring Quorum Failures in Logs

When strong consistency is active, you must monitor server logs for QUORUM_COMMIT and QUORUM_FAIL messages. A spike in QUORUM_FAIL typically indicates network instability or overloaded nodes that cannot respond within the timeout window, leading to application-level write errors.

Mismatching Client Consistency Levels

While the server enforces the consistency mode, clients should be configured to request the appropriate consistency level (e.g., CONSISTENCY_LEVEL_MAJORITY) in their API calls to ensure the client-side logic aligns with the server's behavior.