Memory allocation and mutation consistency limits of sequtils.chunks pagination
0 reputation · 02 Apr 2024, 12:37 UTC
Nim's sequtils.chunks procedure provides a documented way to bound an in-memory sequence into fixed-size sub-sequences, which can act as a simple pagination layer when the dataset fits in RAM. For sequences that approach available heap, however, the practical boundaries of this approach remain unclear.
Because each chunk produced by chunks is a new sequence, repeatedly paginating a very large seq may generate many allocations and increase GC pressure. The standard library also does not appear to document whether mutating the source sequence during chunk iteration is supported, or what consistency guarantees apply if elements are added or removed mid-iteration.
Is there a documented allocation or size threshold beyond which chunks becomes unsuitable for pagination? What consistency guarantees exist when the source sequence is mutated during iteration? Are there standard-library patterns that stream bounded results without materializing each chunk as a separate sequence?