Stop Caching: Aerospike's Hybrid Memory Architecture and the NVMe-as-Memory Decision

The cache trap: when NVMe is just a faster disk

Most teams plug NVMe into a server, format it with a filesystem, and let the operating system page cache absorb reads. That works for general-purpose workloads, but it inserts the kernel into every I/O path. Aerospike's Hybrid Memory Architecture (HMA) starts from a different premise: NVMe is not a disk to be cached, but a byte-addressable extension of RAM.

What HMA actually does

HMA splits the database into two planes. The primary index—the map from record keys to storage locations—lives entirely in DRAM. The record data itself, the blobs, is written directly to raw flash using a log-structured pattern. There is no filesystem metadata, no page cache, and no double buffering. Because the index is in memory, primary-key lookups do not need to consult the device unless the record is not already cached in DRAM.

This is not a block cache. The device is the primary store for data, not a copy of something already on disk.

Configuring a namespace for raw device access

On Aerospike Database 5.x and later, the namespace stanza controls whether data lands on a file inside a filesystem or directly on a block device. The setting is storage-engine.

Run the following edit on the Aerospike server node. You need root privileges or membership in a group that grants read/write access to the target block device. The device must be unmounted and not hold a partition table.

namespace telemetry {
    replication-factor 2
    memory-size 8G
    storage-engine device

    # raw NVMe device, not a partition or file
    device /dev/nvme0n1
}

After saving aerospike.conf, restart the Aerospike service. On a systemd host, that is typically sudo systemctl restart aerospike. The service will claim the device directly; do not mount it.

Risk: switching an existing namespace from file to device changes the on-media layout. Treat it as a destructive change for that namespace. If you must roll back, revert the stanza to the previous file-based configuration and restore data from a snapshot or cross-datacenter replication (XDR) backup.

Checking that the device is in the memory path

Once the node is running, use asinfo to confirm the namespace is using raw device storage and that the index fits in RAM. Run this from any host that can reach the Aerospike node, or locally on the server. Replace <host> and <port> with your access address and service port (default 3000).

asinfo -h <host> -p <port> -v "statistics"

Look for index_memory_used and compare it to the configured memory-size. If index_memory_used approaches the limit, the DRAM ceiling—not the flash capacity—will throttle the namespace. Also inspect disk_utilization to verify the raw device is receiving writes.

For a latency check, run a primary-key read benchmark against the device-backed namespace and compare p99 latency with a file-system namespace of similar size. The HMA path should show tighter tail latency because it avoids kernel cache management, though exact numbers depend on your NVMe IOPS and DRAM speed.

The trade-off: RAM is the hard ceiling

HMA buys predictable latency and lower write amplification by giving up the filesystem safety net. The cost is that every index entry must fit in physical RAM. If the index grows past memory-size, Aerospike cannot spill index entries to flash; performance collapses.

Flash endurance is the second constraint. Because Aerospike writes log-structured records directly to the device, consumer-grade SSDs without power-loss protection or adequate TBW (terabytes written) ratings can fail prematurely. Use enterprise NVMe drives rated for sustained random writes.

Takeaway

Use HMA when your working set index fits comfortably in DRAM and you need consistent, low-latency primary-key access at scale. Configure storage-engine device, verify index_memory_used against your memory-size, and size your NVMe for endurance, not just capacity. If your index is larger than available RAM, a traditional disk-backed architecture with a larger flash footprint may be the safer choice.