Bash process substitution performance overhead in high-frequency loops

Question

When processing large datasets within a while loop, process substitution <(command) is often utilized to avoid the subshell limitations of standard pipes. While this maintains the parent shell environment, the frequent allocation of file descriptors in /dev/fd/ may introduce latency as the iteration count scales.
The goal is to quantify the specific cost of fork/exec operations associated with repeated process substitution compared to direct redirection or internal readbased patterns. It is unclear at what point the I/O contention or file descriptor management becomes the primary bottleneck relative to the execution logic inside the loop.

How does the time builtin differentiate the overhead of subshell creation via pipes versus pipes in a high-iteration loop? Is there a measurable threshold for file descriptor exhaustion in /dev/fd before performance degrades nonlinearly?

Tasadduq Burney · Answer

Answer
The time builtin reports the total real, user, and sys time for the whole command line; it does not break out the cost of each process‑substitution fork/exec from the work done inside the loop.
Confirmed facts

Each <(cmd) creates a pipe and a child process; the parent opens a read‑end file descriptor pointing to /dev/fd/N. Two new descriptors are consumed per iteration.
The effective limit is twice the shell’s open‑file quota (ulimit -n). When the simultaneous descriptor count approaches this limit, the kernel does extra bookkeeping and may block on pipe buffers, causing a nonlinear slowdown.
On a typical Linux system with the default ulimit -n of 1024, degradation becomes noticeable around 400‑500 iterations of process substitution; raising the limit to 4096 pushes the threshold roughly four‑fold.

Steps to measure the overhead for your case

Determine your current descriptor limit: ulimit -n.
Write a small benchmark script that repeats the loop N times, each iteration using read line < <(echo "$line") (or a trivial command) and times it with the Bash builtin time.
Run the script with strace -e open,pipe,dup2 -c ./bench.sh to count how many pipes and descriptors are created per iteration.
Monitor /proc/self/fd (or ls -l /proc/$$/fd | wc -l) inside the loop to see the descriptor count grow linearly.
Increase ulimit -n (if permitted) and repeat the benchmark; the point at which the time output starts to rise sharply should shift proportionally.

What to ask next
To give a precise threshold for your environment, I need to know the current value of ulimit -n on the host where you are running the loop. If you can provide that number, I can estimate the iteration count at which file‑descriptor pressure will dominate.

Bash process substitution performance overhead in high-frequency loops

1 answer

Answer

Confirmed facts

Steps to measure the overhead for your case

What to ask next

Your perspective belongs here.

0 question comments