Designing Memory-Safe Pipelines in Ruby with Enumerator.lazy

You have a log-processing job that reads a few million lines, filters them, transforms the survivors, and writes a summary. The naive Ruby version — lines.map {...}.select {...}.first(100) — builds several intermediate arrays the size of the input before it ever looks at the first result. On a 2 GB input, your process gets killed by the OOM killer long before it prints anything.

The useful takeaway: Enumerator.lazy lets you write the same chained pipeline but evaluate it one element at a time, so memory stays flat regardless of input size. This note covers when that design is appropriate, how to structure it, and where it breaks.

Requirements that justify lazy enumeration

Reach for lazy when most of these hold:

• The input is large, unbounded, or streamed (a big file, an infinite range, a paginated API).
• You only need part of the result — first, take, find, any? — or you consume results one at a time.
• Peak memory matters more than raw throughput.
• The pipeline is a linear chain of map/select/reject/flat_map-style steps.

If the input fits comfortably in memory and you need the whole result anyway, eager enumeration is simpler and usually faster. Laziness adds per-element overhead and defers errors, so it is a trade, not a free win.

The smallest suitable design

Call .lazy once on the source, chain your transformations, and end with exactly one forcing operation (each, first, take(n).to_a, find). Everything between lazy and the forcing call returns another lazy enumerator, so no intermediate arrays are allocated.

# Process a large log file without loading it into memory.
# Run anywhere Ruby >= 2.5 is available; no special permissions needed.
def error_summaries(path, limit: 100)
  File.foreach(path).lazy          # streams lines, never slurps the file
      .map(&:strip)
      .select { |line| line.include?("ERROR") }
      .map { |line| line.split(" ", 3).last }
      .first(limit)                # forcing call: stops after 100 matches
end

Three properties make this design work:

• File.foreach without a block already returns an enumerator, so the file is never fully read.
• first(limit) short-circuits: iteration stops as soon as 100 matches are found, even if the file has millions more lines.
• Each line becomes garbage-collectable after it passes through the chain, so peak memory is roughly one line plus your accumulator.

Trust and data boundaries

Lazy pipelines blur where work happens. Two boundaries deserve explicit attention:

Error surfacing. Exceptions inside a lazy chain raise at the point of forcing, not at the point of definition. If you build the enumerator in one method and force it in another, a malformed line will blow up far from the parsing code. Keep the rescue next to the forcing call, or validate eagerly at the boundary where untrusted data enters.

Captured state. Blocks in the chain are closures. If a block captures a large object (a parsed config, a database row batch), that object stays reachable for the life of the enumerator — which, for an infinite source, is forever. Pass small values into blocks; look things up inside the block rather than capturing big structures.

Operational checks

Verify the behavior rather than assuming it:

• Confirm the Ruby version: RUBY_VERSION must be 2.5 or later; Enumerable#lazy does not exist earlier.
• Confirm laziness in IRB: (1..Float::INFINITY).lazy.select(&:even?).first(3) returns [2, 4, 6] immediately. The eager version would hang forever — a quick proof the chain is lazy.
• Measure memory on a realistic input: compare GC.stat(:heap_live_slots) or /proc/self/status RSS before and after forcing the pipeline, versus the eager equivalent. You should see flat memory for the lazy version and linear growth for the eager one.
• Check for accidental forcing: audit the chain for to_a, count, sort, or reverse in the middle — these materialize everything up to that point.

Failure modes

• Silent materialization. Methods like sort, reverse, group_by, and count cannot be lazy by nature; they consume the whole enumerator. If one sneaks into the middle of the chain, memory spikes exactly as the eager version would.
• Deferred side effects. A map that writes to a socket or database does nothing until the enumerator is forced. Code that builds a lazy pipeline and never fully consumes it silently skips work.
• Non-reusable sources. A lazy enumerator wrapping an IO can only be iterated once. Forcing it twice returns nothing (or raises) the second time.
• Concurrency. Sharing one lazy enumerator across threads without synchronization is undefined behavior; give each thread its own enumerator over the source.

Conditions that would change the design

Revisit the lazy approach if you need the entire result set sorted or grouped (materialization is unavoidable — just use eager code and size the machine accordingly), if the pipeline becomes branching rather than linear (consider a queue-based or actor design), or if profiling shows the per-element lazy overhead dominates on small inputs. For CPU-bound transformation of huge streams, lazy fixes memory but not throughput — that is a parallelism problem, not an enumeration problem.