OutOfMemoryException when paginating a large F# sequence with Seq.skip and Seq.take
0 reputation · 04 Mar 2020, 06:36 UTC
An F# application (targeting a recent .NET, FSharp.Core 8.x assumed) needs to expose paged results from a large dataset represented as a seq<'T>. The intended approach is to bound the query with Seq.skip (page * pageSize) followed by Seq.take pageSize, relying on lazy evaluation so only the requested page is materialized.
Two concerns are unresolved. First, the pipeline currently ends with Seq.toList, and it is unclear whether calling it after Seq.skip/Seq.take is sufficient to avoid loading the full dataset, or whether the conversion must be positioned differently to prevent an OutOfMemoryException on very large sources. Second, Seq.skip appears to iterate skipped elements linearly on non-indexed sources, so deep pages (high page numbers) may become progressively slower.
Is Seq.skip followed by Seq.take and then Seq.toList guaranteed to enumerate only the bounded window, or can the ordering still force full materialization? Does Seq.truncate change this behavior compared to Seq.take when the requested count exceeds available elements? For deep pagination on a lazily generated sequence, is the O(n) cost of Seq.skip unavoidable, and what measurable verification (e.g., timing Seq.skip 10 vs Seq.skip 1000000) confirms it?