Aerospike Hybrid Memory Architecture: Indexes in DRAM, Records on NVMe

Aerospike’s Hybrid Memory Architecture solves the cost-latency trade-off

Many engineering teams managing terabyte-scale NoSQL workloads face a binary choice: provision enough DRAM to keep the entire dataset in memory for sub-millisecond latency, or accept the performance variability of traditional disk-based stores. When your dataset grows beyond a few terabytes, a pure-memory cluster becomes cost-prohibitive, yet standard SSD lookups can introduce latency spikes that break SLAs.

Aerospike’s Hybrid Memory Architecture (HMA) changes the equation. The core idea is simple: indexes live in DRAM, records live on NVMe storage. This decoupling means that every primary-key lookup starts as a memory operation, which resolves to a direct offset on the flash device. Because NVMe drives excel at random reads, the resulting latency stays consistently sub-millisecond, even as the total dataset scales to petabytes.

Why the index-size matters more than the data-size

One of the most counter-intuitive aspects of HMA is how DRAM is sized. In traditional databases, memory requirements often scale with the total data volume. In Aerospike HMA, the index only needs to track unique keys.

# Example: 1 billion records, 1 KB each
# Data stored on NVMe: ~1 TB
# Index entries (64-bit): ~8-12 bytes per record
# DRAM needed for index: ~8-12 GB

In this scenario, you can store a terabyte of data while only allocating roughly 12 gigabytes of RAM for the index. This ratio is what makes HMA cost-efficient: you scale storage independently of memory budget.

NVMe hardware that won’t quit under write load

While the read path benefits from memory-resident indexes, the write path still lands directly on the NVMe device. Aerospike writes to raw block devices, bypassing the filesystem, which means the SSD sees the full volume of database operations.

Using consumer-grade SSDs in an HMA configuration is high-risk. These drives typically lack the endurance (DWPD) and the garbage-collection logic needed for sustained database writes. The result is often a "write cliff": performance degrades sharply once the drive’s internal controller can no longer clear stale blocks fast enough.

To avoid this, Aerospike maintains a Hardware Compatibility List (HCL) of enterprise NVMe drives validated for database workloads. Additionally, over-provisioning—leaving a portion of the drive unallocated—gives the SSD controller headroom for background maintenance without blocking active I/O. A common rule of thumb is 10-30% over-provisioning, depending on the drive model and write intensity.

Verifying your HMA configuration

After provisioning, you should confirm that the cluster is operating in HMA mode and that index lookups are not hitting the device.

# On each Aerospike node, run as root or with sudo
asl -v | grep "index_memory"

Expected check: the output should show index memory usage well within your physical RAM capacity, with room for growth. In your monitoring dashboard, watch for device_wait or elevated I/O latency on the NVMe path. If those metrics trend upward, it can signal that the drive is struggling with garbage collection or is undersized for the write load.

Trade-offs and limitations

• RAM dependency: if the index grows beyond available DRAM, the node cannot serve lookups. Monitor index_memory_used relative to total RAM.
• Cold starts: after a full cluster restart, indexes must be rebuilt from the data on NVMe, which can extend recovery time compared to pure-memory systems.
• HCL lock-in: using drives outside the compatibility list can lead to instability or data integrity risks.

If your workload’s key count grows faster than anticipated, re-size the DRAM allocation in aerospike.conf before the index spills to disk. The configuration change requires a rolling restart, but data on NVMe is preserved.