Groovy 3.0 chunked method forces eager evaluation of lazy iterables

Why chunked() Is Eager

In Groovy 3.0 the chunked(int size) method is implemented by first converting the source iterable to a List (via toList()) and then partitioning that list. For a lazy source such as a large Range or a custom Iterable that generates values on demand, this conversion forces the entire sequence into heap memory before any chunk is produced. The result is an OutOfMemoryError when the source size exceeds available memory.

Should the Standard Library Add a Lazy Variant?

Adding a chunkedLazy method would give users a zero‑copy, stream‑like API and keep memory usage proportional to the chunk size. The trade‑offs are:

• API compatibility – a new method avoids breaking existing code, but developers must discover and adopt it.
• Implementation complexity – a lazy version must return an Iterable that lazily pulls from the source iterator, which is straightforward but requires careful handling of close() for resources like database cursors.
• Performance – lazy chunking adds a small per‑element overhead (iterator calls) compared with the bulk list copy used by chunked(). For very large sources the memory savings outweigh the overhead.

Until such a method exists, developers must implement lazy chunking themselves.

Lazy Chunking Today – Iterator Wrapper

The only guaranteed way to keep the source lazy is to consume its iterator directly and yield fixed‑size batches. The following generator‑style method works on any Iterable and never materialises more than one chunk:

def lazyChunk(Iterable source, int size) {
    def iterator = source.iterator()
    while (iterator.hasNext()) {
        def chunk = []
        int count = 0
        while (iterator.hasNext() && count < size) {
            chunk << iterator.next()
            count++
        }
        yield chunk
    }
}

Because the method uses yield, it must be declared with the @groovy.transform.Generator annotation (or be used inside a closure that is coerced to a Generator). The returned object implements Iterable and can be consumed in a for loop or passed to other lazy operators.

Verification Steps

1. Run (1..10_000_000).chunked(100) and monitor heap with VisualVM – you will see a spike proportional to the range size.
2. Replace the call with lazyChunk(1..10_000_000, 100) and verify that heap stays flat regardless of the range length.
3. If your source is a JDBC ResultSet or similar resource, ensure the iterator wrapper closes the resource when iteration finishes (e.g., by implementing AutoCloseable in the generator).

One missing detail that could change the recommendation: What type of lazy source are you paging (plain Range, database cursor, custom Iterable)? The resource‑cleanup strategy differs for each.