Retry‑safe writes with np.memmap: unresolved durability guarantees

The Short Answer

No, np.memmap.flush() does not provide durability equivalent to fsync(). It ensures that modified pages are pushed from the process memory to the OS page cache, but it does not guarantee that the data has been physically written to stable storage. To achieve durability, you must explicitly invoke OS-level synchronization on the file descriptor.

Explanation of Behavior

NumPy's memmap is a wrapper around the operating system's mmap implementation. When you call flush(), NumPy delegates the operation to the underlying mmap.flush(). In most POSIX and Windows environments, this tells the kernel that the pages are dirty and should be scheduled for writing, but the call typically returns before the disk controller has confirmed the write.

Durability vs. Atomicity

It is critical to distinguish between durability (will the data survive a power loss?) and atomicity (will the write be all-or-nothing?).

• Durability: flush() protects against process crashes (because the OS still holds the data in the page cache), but not against kernel panics or power failures.
• Atomicity: Neither flush() nor fsync() provides atomicity. If a crash occurs during a large write, you may encounter "torn writes," where the file contains a mixture of old and new data.

Implementation for Durable Writes

To guarantee that a retried write will not observe corrupted or missing data, follow these steps:

1. Flush the mapping: Call mm.flush() to move data to the kernel cache.
2. Force Disk Sync: Use os.fsync() on the file descriptor. Since np.memmap does not expose the file descriptor directly in its public API, you must either maintain your own file handle or access the internal mapping: os.fsync(mm._mmap.fileno()).
3. Handle Atomicity: Because in-place writes are not atomic, use the "write-rename" pattern: write to a temporary file, fsync the temp file, and use os.replace() to move it to the target destination.

Verification Command

You can verify that flush() does not trigger a hardware sync by using strace (on Linux) to monitor system calls:

# Run your python script and filter for fsync or msync calls
strace -e trace=fsync,fdatasync,msync python your_script.py

If flush() were providing full durability, you would see an fsync or msync syscall associated with the file descriptor; in standard NumPy usage, you will not.

Assumptions and Constraints

This analysis assumes a standard POSIX-compliant filesystem or NTFS. Note that network filesystems (NFS, SMB) often have weaker fsync and rename semantics, which may render the "write-rename" pattern non-atomic. Additionally, accessing mm._mmap relies on a private attribute that may change in future NumPy versions.