Diagnosing Aerospike Namespace Memory Pressure: Latency Spikes and Evictions

Your Aerospike cluster was running fine, and now p99 read latency has climbed past your baseline while eviction counters start ticking upward. This combination—latency spikes plus evictions—almost always points to namespace memory pressure: the namespace is filling toward its memory-size limit, and the server is evicting records or stalling on defragmentation to stay under it. This guide walks through recognizing the condition, isolating the cause, applying the right fix, and knowing when to escalate.

Recognizing the condition

Memory pressure rarely announces itself with an error. The recognizable pattern is:

• Sustained read/write latency above your baseline (for SSD-backed namespaces, anything persistently above roughly 2ms when you normally see sub-millisecond is worth investigating).
• A non-zero or rising eviction rate when your data model shouldn't be evicting.
• memory-used approaching the configured memory-size for the namespace.
• Log warnings mentioning defragmentation or memory thresholds.

Evictions are the sharpest signal: Aerospike only evicts records (that have no TTL or expired TTL, depending on configuration) when the namespace crosses its high-water mark. If you see evictions on data you expected to keep, memory is the first place to look.

Likely causes and how to tell them apart

Cause	Distinguishing signal
memory-size allocation too small for the dataset	memory-used steadily climbing toward memory-size; evictions begin at HWM
Write throughput outpacing defragmentation	High write-rate, defrag blocked or lagging, device fragmentation warnings in logs
Storage device latency or SSD wear	Elevated device read/write times in statistics; iostat shows high await
Defrag water marks misconfigured	Defrag starts too late (lwm too high) or runs constantly (lwm too low)
Replication factor inflating memory footprint	memory-used roughly scales with replication-factor; per-node usage higher than data size suggests

Ordered diagnostic checks

Run these from any cluster node with asinfo and asadm installed. They are read-only and safe to run in production. Replace <ns> with your namespace name.

1. Check memory headroom. Compare used versus configured memory:

   asinfo -v "namespace/<ns>/memory-used"
   asinfo -v "namespace/<ns>/memory-size"

   If used is above ~80% of size, you are in the pressure zone. Note that in current server versions these statistics may be exposed under different names (e.g., memory_used_bytes); check asinfo -v "namespace/<ns>" for the full list on your version.
2. Check eviction activity.

   asinfo -v "namespace/<ns>/evict-rate"

   Any sustained non-zero rate on data without TTLs confirms active eviction. Compare against your write rate to gauge severity.
3. Check write pressure and queueing.

   asinfo -v "namespace/<ns>/write-rate"
   asinfo -v "namespace/<ns>/pending-txns"

   Growing pending transactions with high write rate suggests the storage layer can't keep up, which feeds back into memory pressure.
4. Inspect device latency. Use asadm -e "show statistics like 'device'" to review per-device read/write times, and cross-check with OS-level tooling:

   iostat -x 5

   Run iostat on the node hosting the namespace (needs the sysstat package). High await or %util near 100 points at the storage layer, not memory sizing.
5. Review defrag water marks.

   asinfo -v "namespace/<ns>/defrag-lwm"
   asinfo -v "namespace/<ns>/defrag-hwm"

   Defragmentation reclaims space from partially-empty write blocks. If defrag-lwm is set too high, defrag starts only when blocks are nearly empty, delaying reclamation; too low and it churns I/O constantly.

Fixes tied to findings

memory-used above 80% of memory-size

Increase memory-size in the namespace stanza of aerospike.conf, or add nodes to spread the data. Before raising it, verify actual free RAM on every node with free -g—sizing past available memory causes swapping, which is far worse than evictions. Apply via a rolling restart, one node at a time, waiting for migrations to complete between nodes.

High eviction rate with saturating write rate

Raise defrag-lwm/defrag-hwm so defragmentation triggers earlier and reclaims blocks faster. For flash storage-engine namespaces, verify post-write-queue is sized adequately for burst absorption. Monitor device utilization after the change—more aggressive defrag increases SSD I/O and wear.

Elevated device latency

Check SSD health and wear indicators (e.g., smartctl -a /dev/<device>, run as root on the node). Consider additional over-provisioning on the drives, moving the namespace to faster storage, or rebalancing data across more devices. If wear is advanced, plan drive replacement before tuning anything else.

Growing pending transactions

Review write-block-size against your record size distribution and consider increasing service-threads if CPU headroom exists. Confirm with top that the node isn't already CPU-saturated before adding threads.

All configuration changes above require a rolling restart to take effect. Confirm the cluster has enough capacity to tolerate one node down at a time before starting.

Verifying the fix

After remediation, re-run the ordered checks and confirm:

• memory-used stabilizes below ~70% of memory-size.
• Eviction rate returns to baseline (near zero, or under 0.5% of write rate if TTL-based eviction is expected).
• Client-side latency (or the Prometheus exporter's latency metrics) returns to your SLA.
• No new defrag or memory warnings appear in the server log.

Capture before/after values for each metric so you can spot the trend recurring as data grows.

When to escalate

Open a support ticket with Aerospike if, after applying the relevant fixes: latency remains above threshold, eviction rate stays above ~1% of write rate, or memory-used still exceeds 95% after a memory-size increase. Attach the server logs, an asadm collectinfo snapshot, your namespace configuration, and a description of the workload profile (record sizes, read/write ratio, TTL usage). These give support what they need without a second round-trip.