Stream‑Slicing in Bash: Using Process Substitution for Clean File Comparisons

Why Do We Need Process Substitution?

When you compare two files with diff, you often have to sort them first so that the order of lines does not affect the result. A naïve approach is to write the sorted output to temporary files, then run diff on those files:

sort file1 > /tmp/s1
sort file2 > /tmp/s2
diff /tmp/s1 /tmp/s2
rm /tmp/s1 /tmp/s2

This pattern introduces extra disk I/O and a cleanup step that can fail if the script aborts. Bash’s process substitution offers a cleaner alternative by letting a command produce a file‑like stream that tools can read directly, without creating physical files.

What Is Process Substitution?

Process substitution is a Bash feature that replaces the syntax <(command) with a pathname that refers to a read‑only file descriptor. The command runs in the background, and its output is made available through a pseudo‑file. For example, diff <(sort file1) <(sort file2) runs two sort processes concurrently and passes their outputs to diff as if they were ordinary files.

Under the hood, Bash opens a new file descriptor for each substitution, writes the command’s output to it, and then passes the path /dev/fd/N to the receiving utility. The /dev/fd interface is supported on most modern Linux, macOS, and BSD kernels, but it is not part of the POSIX standard, so scripts that must run under /bin/sh or dash should avoid this syntax.

Practical Example: Comparing Sorted Files Without Temp Files

Below is a minimal script that demonstrates safe use of process substitution for file comparison. It checks that the running shell is Bash 3.1+ and that /dev/fd is available before proceeding.

#!/usr/bin/env bash
# Exit on any error
set -e

# 1. Verify Bash version
if (( BASH_VERSINFO[0] < 3 || (BASH_VERSINFO[0] == 3 && BASH_VERSINFO[1] < 1) )); then
    echo "Error: Bash 3.1 or newer is required." >&2
    exit 1
fi

# 2. Verify /dev/fd support
if [[ ! -e /dev/fd/0 ]]; then
    echo "Error: /dev/fd is not available on this system." >&2
    exit 1
fi

# 3. Compare two files after sorting
file1=$1
file2=$2

if [[ -z $file1 || -z $file2 ]]; then
    echo "Usage: $0 file1 file2" >&2
    exit 1
fi

# The diff command receives two pseudo‑files created by process substitution.
diff <(sort "$file1") <(sort "$file2")

Key points:

• set -e ensures the script aborts on any command failure.
• The Bash version check uses the BASH_VERSINFO array.
• We test /dev/fd/0 for existence; if it fails, the script exits early.
• Quoting the file names protects against spaces and globbing.
• No temporary files are created; the operating system handles the file descriptors.

Trade‑Offs and Limitations

While process substitution reduces disk I/O and cleanup, it has some caveats:

• Non‑POSIX Compatibility: The syntax is Bash‑specific. Scripts that need to run under /bin/sh or other POSIX shells should use temporary files or mktemp.
• Kernel Support: Some older or embedded kernels may not expose /dev/fd entries. In such environments, the substitution fails with a “bad file descriptor” error.
• Resource Limits: Each substitution opens a new file descriptor. On systems with a very low ulimit -n, large numbers of simultaneous substitutions can exhaust the descriptor pool.
• Read‑Only Streams: The pseudo‑files are read‑only. If you need to write to them, you must use other mechanisms (e.g., named pipes).

How to Verify the Feature on Your System

Before relying on process substitution in a production script, run these quick checks:

1. Bash version: bash --version should report 3.1 or newer.
2. /dev/fd test: echo <(echo test) should output /dev/fd/63 (or similar) without error.
3. Diff test: Create two small files and run diff <(sort file1) <(sort file2). If it runs without error, the feature is operational.

On systems that pass these tests, you can safely use process substitution for lightweight stream handling.

Actionable Takeaway

When you need to feed command output into another utility that expects a filename—such as diff, tar, or awk—consider process substitution to avoid temporary files. Just remember the compatibility and resource constraints discussed above, and guard your scripts with a simple feature test.

Happy scripting!