Processing Datasets Larger Than RAM with NumPy Memory-Mapped Arrays

When your array won't fit in memory

You have a 20 GB numeric dataset and 8 GB of RAM. Loading it with np.load or np.fromfile will either fail or push the machine into swap. The useful takeaway: numpy.memmap lets you index an on-disk array with normal NumPy slicing while the operating system pages only the touched regions into memory. You get out-of-core computation with almost no change to your array code.

A memory map asks the OS to associate a file's bytes with a range of virtual addresses. Reads and writes go through the kernel's page cache, so sequential access is fast and your process's resident memory stays proportional to what you actually touch, not the file size.

Prerequisites

• 64-bit Python with NumPy installed (memmap has been stable for many years; this guide assumes NumPy 1.20+). On 32-bit Python the address space caps mappable size around 2–3 GiB.
• Enough free disk space for the raw data: memmap files are uncompressed, so size equals dtype.itemsize times the element count.
• The exact dtype and shape of the data. Memmap stores no metadata — if you guess wrong, you read garbage silently.

Creating and filling a memmap

Run this in any Python session; no special permissions are needed beyond write access to the target directory. Replace the shape and path with your own.

import numpy as np

path = "matrix.dat"
shape = (10_000_000, 100)   # 1e9 float64 values ≈ 8 GB on disk

arr = np.memmap(path, dtype="float64", mode="w+", shape=shape)

# Fill in chunks so the working set stays small
for start in range(0, shape[0], 100_000):
    stop = min(start + 100_000, shape[0])
    arr[start:stop] = np.random.default_rng(0).standard_normal((stop - start, shape[1]))
    arr.flush()  # checkpoint: force OS write-back for this batch

mode="w+" creates or truncates the file. Other modes: "r" (read-only), "r+" (read/write, file must exist), and "c" (copy-on-write — edits live in memory and never touch disk, useful for scratch work on a file you must not modify).

Check the result before moving on:

import os
expected = np.dtype("float64").itemsize * shape[0] * shape[1]
assert os.path.getsize(path) == expected, "file size mismatch"
assert arr.flags["WRITEABLE"]

Reopening and reading

Because the file holds raw bytes only, every reopen must repeat the dtype and shape. A common pattern is to store that metadata in a small JSON sidecar file.

arr = np.memmap(path, dtype="float64", mode="r", shape=shape)
print(arr[0, :5])          # only the pages covering these rows are loaded
column_mean = arr[::100, 3].mean()   # strided read, still out-of-core

To confirm memory stays bounded, sample RSS (resident set size — the RAM your process actually occupies) while working:

import psutil
rss_mb = psutil.Process().memory_info().rss / 1e6
print(f"RSS: {rss_mb:.0f} MB")   # should be far below the 8 GB file size

If RSS climbs toward the file size, your access pattern is touching too many pages at once — shrink the chunk size or avoid full-array operations like arr.sum(), which stream the whole file (correct, but slow if the data vastly exceeds the page cache). Prefer chunked reductions:

total = 0.0
for start in range(0, shape[0], 100_000):
    total += arr[start:start + 100_000].sum()

Durability: flushing and cleanup

Writes to a memmap sit in OS dirty pages until the kernel writes them back. arr.flush() forces that write at checkpoints you control. When you're done, drop the reference so the mapping closes:

arr.flush()
del arr

NumPy memmap objects also support context managers on recent versions (with np.memmap(...) as arr:), which close the mapping on exit. Two risks to plan around:

• Crash before flush: unwritten pages may be lost. Flush after each logical batch, not just at the end.
• Windows file locking: another process holding the file open can cause PermissionError on open or delete. Ensure previous memmaps are deleted and garbage-collected before reopening with a different mode.

Limitations worth knowing up front

• No compression — the file is raw bytes. If disk space matters, look at HDF5 (h5py) or Zarr, which also carry dtype/shape metadata for you.
• No concurrency control. Two processes writing the same region race silently; use external file locks or partition the file per worker.
• Fixed-size dtypes only. Object arrays and ragged data are not supported.
• Random access across a huge file can thrash the page cache. Sequential or blocked access patterns perform best; if you need random row access at scale, benchmark before committing.

The decision rule: memmap is the right tool when your data is a fixed-type rectangular array, your access is mostly sequential or chunked, and you want minimal code change. Reach for HDF5 or Zarr when you need compression, self-describing files, or concurrent readers.