Using Bash Process Substitution to Stream Data Without Temporary Files

The problem: needing file arguments but wanting to avoid temp files

Many Unix utilities (e.g., diff, join, patch) expect file names on the command line. When the data you want to compare or transform is already produced by another command, the typical workaround is to write intermediate files to disk, then clean them up later. This adds I/O overhead, consumes space, and leaves fragile cleanup logic in scripts.

Thesis: Bash process substitution lets you treat a command’s output (or input) as a file

Process substitution, written as <(cmd) for reading and >(cmd) for writing, expands to a pathname that refers to a pipe or /dev/fd file descriptor connected to cmd. The consumer sees a normal file argument, while the producer streams data directly without creating a persistent file on disk.

How it works

• <(cmd) runs cmd in the background, connects its stdout to a pipe, and returns a name like /dev/fd/63 (or a FIFO under /tmp if /dev/fd is unavailable).
• >(cmd) does the opposite: it connects cmd’s stdin to a pipe and returns a name you can write to.
• The subprocess is asynchronous; the parent shell does not wait for it by default, and its exit status is not placed in $? or PIPESTATUS.

Common practical patterns

• Comparing sorted streams: diff <(sort file1) <(sort file2)
• Tee‑like fan‑out: tee >(gzip > out.gz) >(sha256sum) > /dev/null
• Reading a loop without losing variables: while read -r line; do … done < <(cmd)

Worked example: joining two large CSV files on a key

Suppose you have two CSV files, users.csv (fields: id,name) and orders.csv (fields: order_id,user_id,amount). You want to join them on the user ID without materialising the sorted intermediate files.

# Join on the second column of users.csv and the first column of orders.csv
join -t, -1 2 -2 1 \
    <(sort -t, -k2 users.csv) \
    <(sort -t, -k1 orders.csv)

Each sort process streams its sorted output directly into join via a pipe‑backed file descriptor. No temporary sorted files are created, and the disk I/O is limited to the original CSVs.

Trade‑offs and limitations

• Debugging opacity: Because no intermediate file persists, you cannot inspect the sorted data after the fact. If the command fails, you lose the ability to re‑run just the producer.
• Error handling: The exit status of the subprocess is invisible to the parent. A failing sort inside <(sort …) will not trigger set -e. You must wrap the substitution or check status separately if reliability is required.
• Portability: Process substitution is a Bash‑specific extension (also present in zsh and ksh93). Scripts that start with #!/bin/sh will fail on systems where /bin/sh is dash or ash.
• File‑descriptor consumption: Each <(cmd) or >(cmd) occupies at least one FD in the parent shell. Opening many substitutions in parallel can hit ulimit -n limits.
• SIGPIPE risk: If the consumer closes its end early (e.g., head -n1), the producer receives SIGPIPE and may terminate with status 141 unless the signal is trapped or ignored.

When to use it

Use process substitution when you need to connect a command that expects a file name to another command’s stream, the data volume makes temporary files undesirable, and you control the execution environment (Bash, zsh, or ksh). For portable POSIX scripts or when you need reliable error propagation, fall back to explicit temporary files or a coproc (Bash 4.0+) for bidirectional communication.

As a quick sanity check, you can verify that your Bash supports the feature with:

bash -c 'cat <(echo test)'   # should print "test"

If the command fails, your sh is likely a POSIX‑only shell and you’ll need to adjust the shebang or invoke bash explicitly.