numpy.array() vs numpy.fromiter(): memory‑efficient streaming or fast bulk copy for large iterables?

Question

Goal
Convert a large Python iterable (e.g., a list with >10⁷ elements) to a NumPy ndarray while keeping peak memory usage low and avoiding unnecessary latency.
Constraints and uncertainty
numpy.array() always copies the input, doubling memory temporarily. numpy.asarray() avoids a copy only when the source is already an ndarray, which does not help for plain lists. numpy.fromiter() streams elements without storing the full input, but it requires the total count upfront for optimal allocation and may be slower per element. Version‑dependent optimizations in NumPy 1.26+ can reduce copy overhead, yet the documented default remains copy‑on‑create.
Open questions

Does numpy.fromiter() with a known length consistently use less peak memory than numpy.array() for lists larger than 10⁷ elements?
How does the runtime of fromiter() compare to the bulk copy path on NumPy 1.26+?
Are there cases where pre‑allocating an ndarray and filling it via asarray() or direct assignment outperforms both approaches?

Tasadduq Burney · Answer

Direct answer
If you know the number of elements ahead of time, numpy.fromiter(iterable, dtype, count=N) keeps peak memory close to the size of the resulting array only, while numpy.array(iterable) always makes a temporary copy, roughly doubling memory usage.
For raw speed, numpy.array() is faster because it copies data with a bulk memcpy operation; numpy.fromiter() fills the array element‑by‑element in Python, so its per‑element throughput is lower, and this gap persists in NumPy 1.26+.
Confirmed facts

numpy.array() allocates a new ndarray and copies the entire input, so peak RAM ≈ size(input) + size(output).
numpy.fromiter() streams elements and allocates the output once when the length is supplied, giving peak RAM ≈ size(output) plus a small Python overhead.
The copy path in numpy.array() has not changed in recent NumPy releases; the iteration loop in numpy.fromiter() remains the same.

Practical steps

Determine whether the iterable length is known.
If yes and memory is the primary concern, use numpy.fromiter(it, dtype, count=len).
If speed is more important and you can afford the temporary double memory, use numpy.array(it).
When you need both low memory and high speed, pre‑allocate an empty array (np.empty(N, dtype)) and fill it with a simple loop or np.put; this matches the memory profile of fromiter() while often reaching the speed of array().

Missing diagnostic detail
Do you know the exact length of the iterable before conversion? Knowing this changes whether fromiter() can retain its memory advantage.

numpy.array() vs numpy.fromiter(): memory‑efficient streaming or fast bulk copy for large iterables?

Goal

Constraints and uncertainty

Open questions

1 answer

Direct answer

Confirmed facts

Practical steps

Missing diagnostic detail

Your perspective belongs here.

0 question comments