Stop-writes failure condition in namespace storage engine for repeatable development

Direct Answer

For a single-node Aerospike development environment, the stop-writes engagement parameters are:

• Memory storage engine: memory-high-water and memory-low-water (values in MB or percentage depending on version)
• Device/hybrid storage engine: device-high-water and device-low-water (percentage of configured device capacity)

Persistence mode does not change the threshold evaluation logic, but it determines which metric can trigger stop-writes. Memory-only namespaces respond only to memory pressure; hybrid namespaces may trigger on either memory or device usage, whichever threshold is crossed first.

Pin replication-factor 1 with a single node for repeatable tests. Multi-node emulation does not alter per-node threshold logic, but it changes partition map distribution and client routing, which can make stop-writes appear at different times or on different nodes for the same workload.

Confirmed Parameters by Storage Engine

In-Memory Namespace (storage-engine memory)

namespace test {
  memory-size 256M
  memory-high-water 80   # percentage (6.x/7.x) or MB (5.x)
  memory-low-water 60    # percentage or MB
  storage-engine memory
}

Hybrid/Persistent Namespace (storage-engine device)

namespace test {
  memory-size 128M
  memory-high-water 80
  memory-low-water 60
  storage-engine device {
    file /opt/aerospike/data/test.dat
    filesize 512M
    device-high-water 80   # percentage of filesize
    device-low-water 60
  }
}

Version note: Aerospike 5.x uses absolute MB for memory thresholds; 6.x and 7.x default to percentages of memory-size. Device thresholds have always been percentages of filesize. Check your server version with asd --version before assuming units.

Persistence Mode and Restart Reproducibility

Stop-writes state is ephemeral. On asd restart:

• Memory usage counters reset to zero
• Device usage counters reset (unless flash-cache or device state files retain prior state—see verification)
• The condition must be re-induced by your test load after each restart

Persistence mode affects which resource pressure triggers stop-writes, not whether the condition survives restart. For repeatable tests, script the same load pattern against the same thresholds after every restart.

Single-Node RF=1 vs Multi-Node Emulation

Use a single node with replication-factor 1 for deterministic stop-writes behavior. In this configuration:

• All partitions are master on the single node
• Stop-writes engages based solely on local resource usage
• No partition migration or replica sync traffic affects thresholds

Multi-node clusters (even with RF=1) introduce partition map distribution, client routing via partition awareness, and potential cross-node effects that change when and where stop-writes becomes visible for the same namespace configuration.

Verification Steps for Your Dev Setup

1. Memory stop-writes: Set memory-high-water 60 (or a low MB value in 5.x), write records until the client receives AEROSPIKE_ERR_STOP_WRITES (result code 153), then delete records or wait for eviction to drop below memory-low-water and confirm writes resume.
2. Device stop-writes: Configure a small ramdisk device (e.g., 100MB), set device-high-water 70, write until stop-writes engages, then verify release below device-low-water.
3. Restart reset: After inducing stop-writes, restart asd and confirm usage counters show zero (check asinfo -v 'namespace/test' for memory_used_bytes=0 or device_used_bytes=0). If counters persist, check for flash-cache files or device state retention.

One Missing Diagnostic Detail

What Aerospike server version are you running? The threshold units (MB vs percentage), default values, and whether memory-high-water accepts percentages changed between 5.x, 6.x, and 7.x. This changes the exact configuration syntax and default behavior for repeatable tests.