Question
Bash process substitution performance overhead in high-frequency loops
Neel Lane
0 reputation · 18 Apr 2024, 17:38 UTC
43.6K views
Loading…
0 reputation · 18 Apr 2024, 17:38 UTC
<(command) is often utilized to avoid the subshell limitations of standard pipes. While this maintains the parent shell environment, the frequent allocation of file descriptors in /dev/fd/ may introduce latency as the iteration count scales.The goal is to quantify the specific cost of fork/exec operations associated with repeated process substitution compared to direct redirection or internal readbased patterns. It is unclear at what point the I/O contention or file descriptor management becomes the primary bottleneck relative to the execution logic inside the loop.
How does the time builtin differentiate the overhead of subshell creation via pipes versus pipes in a high-iteration loop? Is there a measurable threshold for file descriptor exhaustion in /dev/fd before performance degrades nonlinearly?
33775 reputation · 18 Apr 2024, 22:03 UTC
The time builtin reports the total real, user, and sys time for the whole command line; it does not break out the cost of each process‑substitution fork/exec from the work done inside the loop.
bash<(cmd) creates a pipe and a child process; the parent opens a read‑end file descriptor pointing to /dev/fd/N. Two new descriptors are consumed per iteration.ulimit -n). When the simultaneous descriptor count approaches this limit, the kernel does extra bookkeeping and may block on pipe buffers, causing a nonlinear slowdown.ulimit -n of 1024, degradation becomes noticeable around 400‑500 iterations of process substitution; raising the limit to 4096 pushes the threshold roughly four‑fold.ulimit -n.read line < <(echo "$line") (or a trivial command) and times it with the Bash builtin time.strace -e open,pipe,dup2 -c ./bench.sh to count how many pipes and descriptors are created per iteration./proc/self/fd (or ls -l /proc/$$/fd | wc -l) inside the loop to see the descriptor count grow linearly.ulimit -n (if permitted) and repeat the benchmark; the point at which the time output starts to rise sharply should shift proportionally.To give a precise threshold for your environment, I need to know the current value of ulimit -n on the host where you are running the loop. If you can provide that number, I can estimate the iteration count at which file‑descriptor pressure will dominate.
Use comments to ask for clarification. Post a solution as an answer.
5,860 reputation · 18 Apr 2024, 21:31 UTC
Each <(command) expansion causes Bash to fork a subshell and exec the command; the file descriptor that points to /dev/fd/N is merely a conduit for the subshell’s output. The descriptor is allocated and then closed within the same iteration, so the kernel’s FD‑allocation overhead is small compared to the cost of the clone and execve system calls. Consequently, the dominant latency in a tight loop comes from the repeated fork/exec pair, not from exhausting the /dev/fd pool unless the subshells are left open (e.g., backgrounded) or the per‑iteration work is extremely short (< ≈ 20 µs).
To isolate this, run the loop under strace -f -e trace=clone,execve,openat,close and count the clone and execve events per iteration; they will match the number of process substitutions, while the openat/close count will be roughly twice that (one for the pipe/fd, one for its close). If you replace the substitution with a simple redirection (read line < file) you will see the clone/execve disappear, confirming that the subshell creation is the primary overhead.