NumPy memmap: Out‑of‑Core Array Processing Guide

Problem

When a dataset is larger than the RAM of a workstation, loading it with NumPy raises MemoryError. The goal is to treat a file on disk as a NumPy array so that read and write operations happen out‑of‑core, using only the pages that are accessed.

Prerequisites

• NumPy version ≥ 1.16 (stable memmap API).
• A filesystem that supports memory‑mapped files (ext4, XFS, NTFS, APFS).
• Enough free disk space for the array file (size = dtype.itemsize × np.prod(shape) + offset).
• Write permission on the directory where the file will be created.

Procedure

1. Create or open a memmap. Use np.memmap(filename, dtype, mode, shape, offset).

   # Example: 10 GB float32 array, start at file beginning
   filename = '/tmp/large_array.dat'          # path to the backing file
   dtype    = 'float32'                       # 4 bytes per element
   shape    = (2_500_000_000,)                # 10 GB = 4 bytes × 2.5 billion
   offset   = 0                               # start at file beginning
   arr = np.memmap(filename, dtype=dtype, mode='w+', shape=shape, offset=offset)
   

   • Use mode='w+' for a new file (read‑write).
   • Use mode='r+' for an existing file you intend to modify.
   • Use mode='r' for read‑only access.
2. Access data in slices. Only the pages touched by a slice are loaded into RAM.

   # Write a single element at position 1 000 000
   arr[1_000_000] = 1.0
   

   When you later read arr[1_000_000:1_000_001], the OS pages containing that element are mapped, but the rest of the 10 GB array stays on disk.

3. Force dirty pages to disk. Call arr.flush() to ensure modifications are written before the file handle is closed.

   arr.flush()  # optional; context manager does this automatically
   

4. Release resources safely. Either delete the array object or use a context manager (NumPy ≥ 1.24).

   del arr               # manual release
   # or
   with np.memmap(filename, dtype=dtype, mode='r+', shape=shape) as arr:
       # work with arr here
   

Verification Checks

• File size. After creation, the file size should equal dtype.itemsize * np.prod(shape) + offset. Verify with the OS (e.g., ls -lh or stat).
• Write‑ability flag. Inspect arr.flags['WRITEABLE']; it must match the intended mode (True for 'w+' or 'r+', False for 'r').
• Memory footprint. While accessing slices, monitor RSS with psutil.Process().memory_info().rss. It should stay close to the working set size, not the full array size.
• Crash recovery. Kill the writer process (e.g., kill -9) and reopen the file with identical dtype, shape, and offset. Confirm that previously flushed values persist.

Recovery and Limitations

• If the process crashes before flush(), the file remains valid; simply reopen it with the same parameters.
• To protect against corrupted flushes, keep a checksummed copy of the file or use mode='c' (copy‑on‑write) to isolate modifications.
• On Windows, ensure no other process holds the file open before reopening; otherwise a ValueError may be raised.
• Concurrent writes from multiple processes to the same memmap region cause data races. Use file locking or separate files per process.
• Offset must be a multiple of the system page size (typically 4096 bytes). Misaligned offsets raise ValueError on some platforms.
• Memory‑mapped files count against virtual memory limits (e.g., ulimit -v). Very large arrays may hit this limit even though RSS stays low.

When to Prefer memmap Over Chunking

Use np.memmap when:

• Access patterns are sequential or strided and the working set fits in the OS page cache.
• The dataset is larger than RAM but the OS can page in only the needed portions.

Avoid memmap for:

• Tiny arrays (< 10 MB) where the overhead of mapping outweighs any benefit.
• Random‑small‑access workloads that cause frequent page faults, which can degrade performance.

Summary

Memory‑mapped arrays let you treat disk‑backed data as if it were in RAM, enabling out‑of‑core computation without exhausting physical memory. By following the steps above, verifying file size, write‑ability, and memory usage, and using proper cleanup, you can safely process datasets that exceed your workstation’s RAM.