Process Substitution in Bash: A Practical Guide to Stream‑Based Composition

Why You Should Care About Process Substitution

When you need to feed the output of one command into another without creating a physical file, Bash’s <(cmd) (input) and >(cmd) (output) syntax gives you a clean, efficient solution. It’s especially handy for one‑liners that would otherwise require temporary files or named pipes you would have to clean up manually.

What Exactly Is Process Substitution?

Process substitution creates an anonymous file descriptor that points to the stdout or stdin of a subshell running cmd. The outer command receives a path like /dev/fd/63 or a FIFO name, which it can read from or write to as if it were a regular file. The key points are:

• It runs cmd in a child subshell, so any variable changes or traps inside that subshell don’t affect the parent.
• The file descriptor is closed when the subshell terminates, but the parent may finish earlier; the kernel buffers data until the reader consumes it.
• Process substitution is a Bash (and ksh93, zsh) extension; it is not POSIX and will fail in sh or dash.

Typical Use Cases

Below are the patterns that most people use, along with a short explanation of why they work.

1. Comparing Sorted Streams

Instead of creating file1.sorted and file2.sorted, you can sort on‑the‑fly:

diff <(sort file1) <(sort file2)

The sort commands run in parallel, and diff reads from the two temporary descriptors.

2. Joining or Pasting Lines

When you want to combine two files line‑by‑line, join and paste accept process‑substituted arguments:

join <(sort a.txt) <(sort b.txt)
paste <(cat a.txt) <(cat b.txt)

3. One‑Pass Tee to Multiple Consumers

Send the same output to several commands without re‑running it:

tee >(gzip > out.gz) >(sha256sum > out.sha) > out.txt

Here tee writes to three destinations: a file, a gzip process, and a sha256sum process.

Under the Hood: How Bash Implements It

When Bash sees <(cmd) or >(cmd), it performs the following steps:

1. Creates a new file descriptor (e.g., /dev/fd/63) that points to a FIFO or a file‑descriptor backed by /proc/self/fd.
2. Spawns a subshell to run cmd, connecting its stdout (for <(cmd)) or stdin (for >(cmd)) to that descriptor.
3. Substitutes the path (or the numeric descriptor) into the outer command’s arguments.
4. Closes the descriptor in the parent when the outer command finishes; the kernel cleans up the FIFO automatically.

Because the subshell runs asynchronously, the parent may finish before the child does. Bash only waits for the child when the descriptor is closed, which can lead to subtle timing issues.

Portability and Limitations

Process substitution is not part of the POSIX specification. If you write a script that might be executed by /bin/sh on a system that uses dash or busybox sh, the syntax will cause a syntax error. To guard against this, always start scripts with:

#!/usr/bin/env bash

or use ksh93 or zsh if you need the feature on a system that lacks Bash.

Other practical pitfalls include:

• Exit status loss: The exit code of the inner command is not propagated to the outer command. For example, diff <(false) <(echo ok) will still exit with 0 even though false failed. Work around this by capturing the PID of the inner command with $! and wait explicitly.
• Deadlock risk: If the inner command produces more data than the kernel pipe buffer (typically 64 KiB) and the outer command reads slowly, the writer can block. This is rarely an issue for small streams but can surface in high‑throughput pipelines.
• Cleanup on crash: On older systems where /dev/fd is unavailable, Bash falls back to creating a named FIFO in $TMPDIR. If the parent process dies before the FIFO is read, the FIFO may linger and consume space.
• Nested depth: Each nested <(cmd) adds another subshell. Excessive nesting can hit the shell’s recursion limit and make debugging harder.

Concrete Example: Diffing Large Logs Without Temp Files

Suppose you have two log files, app1.log and app2.log, each sorted by timestamp. You want to find differences between them efficiently.

#!/usr/bin/env bash
set -euo pipefail

# Capture the PIDs of the sorting processes so we can check their exit status.
left_sort_pid=$(sort app1.log > /dev/null && echo $! | tr -d '\n')
right_sort_pid=$(sort app2.log > /dev/null && echo $! | tr -d '\n')

# Run the diff with process substitution.
diff <(sort app1.log) <(sort app2.log)

# Wait for the sorting processes and report any failures.
wait $left_sort_pid || echo 'Left sort failed'
wait $right_sort_pid || echo 'Right sort failed'

In this script we:

• Run sort in the background to capture its PID.
• Use process substitution for the actual diff.
• Explicitly wait for the sort processes to surface any errors.

Running the script with shopt -s extdebug shows that the subshells are separate processes, confirming isolation.

When to Use It and When to Avoid It

Use process substitution when:

• You need to avoid writing temporary files for speed or disk space.
• All target environments are Bash, ksh93, or zsh.
• The data size is moderate and you’re not worried about pipe buffer limits.

Avoid it when:

• Your script must run under /bin/sh or a POSIX‑only shell.
• You need reliable error propagation from the inner command.
• There is a risk of deadlock due to large data streams.

Actionable Takeaways

1. Prefix scripts with #!/usr/bin/env bash if you rely on process substitution.
2. When exit status matters, capture $! and wait on the inner command’s PID.
3. Test on a minimal sh environment to confirm portability before shipping.
4. For large data pipelines, consider using named pipes manually so you can monitor buffer usage and cleanup explicitly.
5. Document the use of process substitution in your README so other maintainers understand the shell requirement.

Process substitution is a powerful tool that, when used judiciously, can make your Bash scripts cleaner and faster. Just be mindful of its non‑POSIX nature and the subtle exit‑status and buffering nuances that can trip up even seasoned scripters.