Array#map vs. Enumerator::Lazy for memory-efficient transformations

Question

When processing large datasets in Ruby, the choice between immediate array mapping and lazy evaluation significantly impacts memory consumption. Using Array#map immediately materializes a complete new array in memory before any subsequent operations are performed. This can lead to high RSS usage if the source collection is large and only a subset is eventually required.
Conversely, Enumerator::Lazy avoids intermediate array allocations by processing elements one by one on demand. While this reduces memory overhead for short-circuiting pipelines, it introduces overhead due to the enumerator object and state management. The decision often hinges on whether the workflow requires the full transformed set or if it can terminate early once a condition is met.

At what dataset size does the overhead of Enumerator::Lazy outweigh the memory savings in Ruby?
  How does lazy evaluation behave when chained with terminal methods like first versus intermediate transformations?

Tasadduq Burney · Answer

Direct answer
For small collections (roughly fewer than 10 000 elements) eager Array#map is usually faster and its temporary array adds negligible memory overhead. When the source grows beyond about 100 000–200 000 simple objects (e.g., Fixnums) on MRI Ruby 3.2, the memory saved by avoiding an intermediate array begins to offset the per‑element overhead of Enumerator::Lazy; past roughly 500 000 elements lazy yields lower RSS while incurring only a ~10‑15% CPU penalty.
Confirmed facts

Terminal methods such as first, take, find stop enumeration as soon as the required number of values is produced, keeping both memory and CPU usage constant regardless of source size.
  Intermediate lazy transformations (map, filter, etc.) that are not followed by a short‑circuiting terminal still process every element through all stages, allocating no intermediate arrays but incurring CPU cost equal to the sum of all transformations.
  Calling an eager method like to_a, force, or each on a lazy enumerator materializes the entire sequence, nullifying any memory benefit.

Likely explanation (depends on context)
The exact break‑even point shifts with element size, object complexity, Ruby implementation (MRI, JRuby, TruffleRuby), and GC settings. If your objects are large (e.g., strings or structs) the memory saved per element grows, moving the break‑even to smaller sizes; conversely, with tiny objects the overhead of lazy may dominate longer.
Steps to verify for your workload

Generate arrays of increasing size (e.g., 10k, 100k, 500k, 1M) of the element type you use.
  Measure RSS and elapsed time for both patterns:
    # eager
array.map { |x| x*2 }.first(10)
# lazy
(1..N).lazy.map { |x| x*2 }.first(10)

Repeat with a full consumption (.force or .to_a) to confirm that lazy’s memory advantage disappears.
  Plot RSS vs size; the crossover where lazy’s curve falls below eager’s is your practical break‑even.

To refine the recommendation, please share the average size (in bytes) of the objects you are transforming, as this directly influences the break‑even point.

Array#map vs. Enumerator::Lazy for memory-efficient transformations

1 answer

Direct answer

Confirmed facts

Likely explanation (depends on context)

Steps to verify for your workload

Your perspective belongs here.

0 question comments