Zsh Array Slicing: Memory Overhead During Large Dataset Pagination
27.5K reputation · 24 Dec 2020, 06:55 UTC
Zsh 5.x provides native array slicing via the array[start,end] syntax, which allows for bounding datasets returned by command substitutions without relying on external utilities like less. This mechanism is often used to implement windowed data views within shell scripts.
When managing exceptionally large datasets stored in memory, there is a potential performance trade-off between using standard parameter expansion and the zsh/parameter module for slicing. Specifically, it is unclear how the shell handles memory allocation when creating slices of arrays that contain hundreds of thousands of elements, especially when combined with glob qualifiers for filtering.
- Does Zsh perform a shallow copy or a deep copy when slicing a large array into a smaller subset?
- At what threshold of array size does the latency of internal slicing exceed the overhead of piping to an external paginator?
zsh1 answer
1 question comment
Use comments to ask for clarification. Post a solution as an answer.
27,525 reputation · 24 Dec 2020, 11:52 UTC
Even when you immediately unset a sliced array, the memory used for that slice may not be returned to the operating system right away. Most malloc implementations keep freed blocks in their internal pools for reuse, so the resident set size (RSS) observed with ps -o rss= can stay elevated across many pagination iterations, giving the impression of a leak.
This effect is independent of how the slice is created: arr[start,end], ${arr[@]:offset:length}, or assigning via typeset -a slice=${arr[@]:offset:length} all result in a fresh copy of the selected elements. The same holds for negative indices (e.g., arr[-10,-1]) and for associative arrays – slicing the values still copies the scalar data.
If you need to keep memory usage truly constant, avoid materialising a slice altogether. Iterating over indices with a for (( i = start; i <= end; i++ )) loop or using while read -r line; do ... done < <(printf '%s\n' "${arr[@]}") (where the process substitution reads the original array without copying) prevents the allocator from allocating new blocks for each page.