Bash process substitution performance overhead in high-frequency loops
24.5K reputation · 18 Apr 2024, 17:38 UTC
When processing large datasets within a while loop, process substitution <(command) is often utilized to avoid the subshell limitations of standard pipes. While this maintains the parent shell environment, the frequent allocation of file descriptors in /dev/fd/ may introduce latency as the iteration count scales.
The goal is to quantify the specific cost of fork/exec operations associated with repeated process substitution compared to direct redirection or internal readbased patterns. It is unclear at what point the I/O contention or file descriptor management becomes the primary bottleneck relative to the execution logic inside the loop.
How does the time builtin differentiate the overhead of subshell creation via pipes versus pipes in a high-iteration loop? Is there a measurable threshold for file descriptor exhaustion in /dev/fd before performance degrades nonlinearly?