np.memmap vs np.load mmap_mode for writable large arrays under memory pressure

Short answer

Use np.memmap with mode 'r+' (or 'w+') and call flush() at explicit checkpoints. This gives deterministic control over when dirty pages reach the file system. np.load(..., mmap_mode='c') relies on copy‑on‑write and the interpreter’s garbage collector to write back changes, which is nondeterministic and can consume extra memory under pressure.

Neither API documents a guarantee that data survives an unclean shutdown; durability depends on OS page‑cache behavior and whether a flush or deallocation occurs before the crash.

Why np.memmap is more predictable

• Direct mapping: writes go straight to the OS page cache for the mapped file.
• Explicit flush() forces the OS to write dirty pages; you decide when.
• No hidden private copy, so memory usage stays at the mapped size.

Behavior of np.load(mmap_mode='c')

• Creates a copy‑on‑write mapping; modifications are kept in a private anonymous buffer.
• Changes are written back only when the array object is deallocated or flush() is called on that array.
• Deallocation timing depends on reference counting / GC, making persistence timing unpredictable.
• The private buffer can increase resident memory, worsening pressure.

Crash resilience

Both approaches leave data in the OS page cache until a flush or sync occurs. An abrupt process termination (SIGKILL, power loss) can lose any unflushed pages. For stronger durability you must call os.fsync(fd) on the underlying file descriptor after flush().

Practical verification steps

1. Create a test file: arr = np.memmap('test.dat', dtype='float64', mode='w+', shape=(1000,))
2. Modify an element, call arr.flush(), then os.fsync(arr.fileno()).
3. Kill the interpreter immediately after step 2 and re‑open the file to confirm the change persisted.
4. Repeat with np.load('test.dat', mmap_mode='c'), modify, let the array go out of scope (or call flush()), then crash; observe whether the write appears.

One missing diagnostic

Does the application already call flush() (or os.fsync) before normal termination? If not, adding explicit flush points to np.memmap is the only way to get predictable persistence.