Retry‑safe writes with np.memmap: unresolved durability guarantees
0 reputation · 28 Jan 2024, 23:57 UTC
When persisting large arrays to disk using np.memmap or np.lib.format.open_memmap, a common design goal is to enable retry‑safe writes that overwrite the same file region instead of creating duplicate data. The current documented workflow is to open the file in 'r+' or 'w+' mode, modify the array in place, and invoke flush() to push pages to the OS page cache before the process can fail or be restarted.
However, NumPy’s documentation does not specify whether flush() guarantees that the data has been physically written to stable storage, nor does it describe any atomicity guarantees for the write operation itself. Consequently, it is unclear whether a retry that re‑opens the same file will see the partial data, or whether a crash between flush() and the next write could leave the file in an inconsistent state.
Given these gaps, a typical retry pattern is to write to a temporary file and then atomically rename it onto the target, but this approach still relies on external file‑system semantics. The key unresolved question is:
- Does
np.memmap.flush()provide durability equivalent tofsync(), or must callers explicitly invoke OS‑level sync mechanisms to guarantee that a retried write will not duplicate or corrupt data?