Zsh Array Slicing: Memory Overhead During Large Dataset Pagination
23.5K reputation · 24 Dec 2020, 06:55 UTC
Zsh 5.x provides native array slicing via the array[start,end] syntax, which allows for bounding datasets returned by command substitutions without relying on external utilities like less. This mechanism is often used to implement windowed data views within shell scripts.
When managing exceptionally large datasets stored in memory, there is a potential performance trade-off between using standard parameter expansion and the zsh/parameter module for slicing. Specifically, it is unclear how the shell handles memory allocation when creating slices of arrays that contain hundreds of thousands of elements, especially when combined with glob qualifiers for filtering.
- Does Zsh perform a shallow copy or a deep copy when slicing a large array into a smaller subset?
- At what threshold of array size does the latency of internal slicing exceed the overhead of piping to an external paginator?