NumPy .npy files and memory-mapped storage interoperability

Interoperability of Memory-Mapped .npy Files

When using np.load(filename, mmap_mode='r'), NumPy does not load the entire dataset into RAM. Instead, it reads the .npy header to determine the array's shape, dtype, and byte order, then creates a numpy.memmap object that maps the file's data segment directly into the process's virtual address space.

Header Version Mismatches

NumPy handles header versioning during the initial call to np.load(). The .npy format is designed for backward compatibility; newer versions of NumPy can typically read files created by older versions. If a file is opened in mmap_mode, the header is parsed fully before the memory map is established.

• Backward Compatibility: If the file version is older than the current NumPy installation, the library applies the necessary logic to interpret the header and map the data correctly.
• Forward Compatibility: If the file was created with a version of NumPy newer than the one attempting to load it, np.load() will typically raise a ValueError or OSError during the header parsing phase, before the memory map is created.

External File Modifications

The behavior of a memory-mapped array when the underlying file is modified by an external process depends on the Operating System's page cache and the mmap_mode used:

• Data Consistency: Because memmap leverages the OS kernel's page cache, changes made by another process to the file on disk are generally reflected in the memory-mapped view once the OS synchronizes the pages. However, this is not atomic.
• Corruption Risks: If an external process modifies the header of the .npy file while it is mapped, the existing NumPy array object will not update its shape or dtype attributes, leading to a misalignment between the array's metadata and the actual bytes on disk. This often results in corrupted data reads or segmentation faults.
• File Truncation: If an external process truncates the file to a size smaller than the mapped region, accessing the missing segments will trigger a Bus Error (on Unix) or an access violation (on Windows).

Verification Steps

To verify the memory-map status of your array, check the .base attribute:

import numpy as np

# Load with memory mapping
data = np.load('large_dataset.npy', mmap_mode='r')

# Verify it is a memmap object and not a standard ndarray
print(type(data.base))
# Expected output: <class 'numpy.memmap'>

Diagnostic Detail Needed: Are you accessing these files across a network filesystem (e.g., NFS, SMB) or a local disk? Network filesystems often have different caching behaviors that can delay or prevent the visibility of external modifications in a memory-mapped view.