Limits of Data.Array.slice for paginating large datasets after full JSON decode

Direct Answers

1. Data.Array.slice allocates a new array containing only the requested elements. Memory allocation is proportional to the slice size, even though the source array is already in memory. This is a copy operation, not a view.
2. Data.List.take/drop does not reduce memory after the full Array exists. Converting an Array to a List wraps each element in a linked-list node (Cons), adding allocation overhead and increasing garbage collection pressure. The original Array remains in memory until garbage collected.
3. No built-in incremental JSON decoding exists in standard PureScript libraries (argonaut, simple-json, foreign). They require the complete JSON string or buffer before parsing. To avoid full materialization, you need a streaming parser or a custom decoder that processes chunks.

Why This Happens

PureScript Arrays compile to JavaScript Arrays. Data.Array.slice maps to Array.prototype.slice, which creates a new dense array with copied references. The source array stays alive as long as any reference exists (including the affjax response handler's closure).

Converting to Data.List via fromFoldable or toList builds a linked structure: each element becomes a Cons cell with a pointer to the next. For a 10,000-element page, that's 10,000 additional heap objects.

Practical Verification

-- Profile in Chrome DevTools Memory tab:
-- 1. Fetch 100k records via affjax
-- 2. Take heap snapshot
-- 3. Run: slice 0 1000 largeArray
-- 4. Take second snapshot
-- 5. Compare retained size delta

Expect the delta to match roughly sliceLength * pointerSize (8 bytes per reference on 64-bit V8) plus array header overhead.

Workarounds Within Current Constraints

• Server-side pagination: Request only the page you need (offset/limit or cursor-based). Eliminates client-side slicing entirely.
• Virtualized rendering: If the dataset must stay client-side, render only visible rows (e.g., with purescript-virtualized or similar). The full array stays in memory but DOM nodes stay minimal.
• Web Workers: Move the full array and slicing logic to a worker; transfer only the page slice via postMessage (structured clone copies, but keeps main thread responsive).

Incremental Decoding Options (Requires Library Change)

• argonaut / simple-json: No streaming API. Full parse required.
• Custom decoder: Write a parser using purescript-parsing or purescript-attoparsec that consumes a ReadableStream chunk-by-chunk, emitting decoded records as they complete. Significant effort; only justified for very large payloads.
• JSON Lines / NDJSON: If the API can return newline-delimited JSON, each line parses independently. Fetch as text stream, split lines, decode per line.

One Missing Diagnostic Detail

Which JSON decoding library are you using (argonaut, simple-json, foreign-generic, other)? The answer determines whether any experimental streaming support exists or if a custom parser is the only path.