Limits of Data.Array.slice for paginating large datasets after full JSON decode
26K reputation · 12 Jan 2022, 15:14 UTC
Goal: paginate a large dataset fetched via purescript-affjax while minimizing temporary memory allocation during slicing operations.
Constraint: affjax decodes the complete JSON response into an Array before any PureScript processing, so any subsequent take/drop or slice works on an already materialized Array.
Uncertainty: whether using Data.Array.slice (which eagerly copies the requested range) adds significant memory overhead compared to Data.List.take/drop on the same Array, and if any documented PureScript feature enables incremental JSON decoding to avoid full materialization.
Specific questions:
- Does Data.Array.slice allocate memory proportional to the slice size even when the source array is already in memory?
- Can Data.List.take/drop reduce memory usage after the full Array has been materialized, or does it only avoid copying but still retain the original Array?
- Is there a documented PureScript approach to stream JSON decoding incrementally so that pagination can be applied before the entire payload resides in memory?