Load Large NumPy Arrays Without Filling RAM

Problem

Your script needs one slice from a multi-gigabyte NumPy array saved on disk, but np.load() reads the entire file into memory before you can index it. On a machine with limited RAM, that load can trigger heavy swapping or kill the process before you ever see the data.

Useful takeaway

Open the file with np.load(path, mmap_mode="r"). NumPy maps the file into your process's virtual address space and loads only the disk pages—fixed-size blocks of memory—the code actually touches, so reading a small slice stays cheap even when the file is far larger than available memory.

Prerequisites

• NumPy installed in the active Python environment (concepts apply to the 1.x and 2.x release series).
• A .npy file created with np.save().
• Read permission on the file; write permission on the directory if you are creating the file.

Check the installed NumPy version

Run this in a terminal:

python -c "import numpy; print(numpy.__version__)"

No special permissions are required. The command prints a version string. If it fails with ModuleNotFoundError, install NumPy before continuing.

Save an array to disk

Run this in a Python script or REPL:

import numpy as np

big_array = np.arange(100_000_000, dtype=np.float64)
np.save("", big_array)

Replace <path/to/dataset.npy> with a writable path. An existing file at that path is overwritten without warning.

Load the array without reading it all into RAM

Run this in Python:

mapped = np.load("", mmap_mode="r")

With mmap_mode="r", NumPy returns a memory-mapped array. The file remains on disk, and the operating system brings in only the pages your code touches.

Confirm the mapping worked

print(mapped.shape, mapped.dtype)
print(type(mapped).__name__)

Shape and dtype print immediately without reading the data payload. The type name is memmap. Reading a single element or a small slice should not cause a large jump in process memory.

Read a slice

first_row = mapped[0, :]
print(first_row[:5])

Only the bytes for the requested slice are brought into memory. Scanning the whole array eventually pages in most of the file, but random access to a small region remains inexpensive.

Modify values in place

Use mmap_mode="r+" when you need to write values back to the file:

rw = np.load("", mmap_mode="r+")
rw[0, 0] = -1.0

Changes are written directly to the .npy file on disk. There is no automatic rollback. Keep a backup copy before mutating a large mapped array, or write to a temporary copy and replace the original only after verification.

Comparison: default load versus memory mapping

Loading method	RAM behavior	Can modify	Writes to disk	Best for
np.load(path)	Reads entire array into RAM	Yes, in memory only	No	Small arrays that fit in memory
np.load(path, mmap_mode="r")	Loads accessed pages only	No	No	Read-only exploration of large arrays
np.load(path, mmap_mode="r+")	Loads accessed pages only	Yes	Yes, directly to the .npy file	Large arrays needing targeted in-place fixes

Limitations

• mmap_mode applies to .npy files, not .npz archives.
• Read-only mode rejects assignments; attempting to assign to a mapped array opened with "r" raises an error.
• With "r+", there is no undo. A crash during a write can leave the file partially modified.
• Accessing widely scattered regions can trigger many page faults and slow disk I/O.
• NumPy 2.0 deprecated legacy aliases such as np.float. Specify explicit dtypes like np.float64 when creating arrays.

Practical verification

1. Save a modest array to <path/to/test.npy>.
2. Load it with mmap_mode="r".
3. Record process memory before and after reading a small slice, using your OS task manager or a tool such as psutil.
4. Confirm memory stays essentially flat and the slice returns correct values.
5. If you test "r+", back up the file first and confirm the change appears in the file on disk after closing Python.

Recovery

If a read-write mapped file is corrupted by a bad script, restore it from the backup made before mutation, or regenerate it from the source data. Memory mapping does not provide transactions or version history.