Choosing Between Process Substitution, Named Pipes, and Temporary Files in Bash

The Problem: Moving Data Between Processes in Bash

Bash scripts frequently need to pass data between commands without writing to disk. Three mechanisms exist: process substitution (<() >()), named pipes (mkfifo), and temporary files (mktemp). Each has different constraints around portability, seekability, cleanup, and concurrency. The wrong choice leads to deadlocks, leaked files, or scripts that fail on minimal containers.

Takeaway: For Bash 4.0+ scripts where seekability isn't needed, default to process substitution. Use temporary files with mktemp and trap cleanup for POSIX portability or when random access is required. Reserve named pipes for long-running co-processes or multiple independent readers.

Constraints and Supported Options

The table below compares the three mechanisms across the criteria that matter for production scripts.

Criterion	Process Substitution	Named Pipe	Temporary File
Portability	Bash, ksh, zsh	POSIX	POSIX
Seekable	No	No	Yes
Auto-cleanup	Yes	No	No
Multi-reader	No	Yes	Yes
Bidirectional	No	Yes	Yes
Disk I/O	Kernel pipe	Kernel pipe	Physical
Concurrency	High	Medium	High

Trade-offs in Detail

Process Substitution

Process substitution creates an anonymous pipe or /dev/fd/N entry. The inner command runs in a subshell; its stdout (for <()) or stdin (for >()) is connected to a file descriptor passed to the outer command. No filesystem entry is created, so cleanup is automatic when the outer command finishes reading.

Limitations: the file descriptor is not seekable. The subshell exits when the outer command closes the FD, which means signals like SIGINT sent to the main script may not reach the subshell unless explicitly forwarded with trap. In Bash < 4.4, process substitution inside command substitution $( <(...) ) fails due to a parsing bug.

On systems without /dev/fd (rare, some minimal containers), Bash falls back to creating a named pipe transparently. This changes the visible FD numbers but not the semantics.

Named Pipes

mkfifo creates a persistent filesystem entry. Multiple processes can open it independently, enabling bidirectional communication and multiple readers. However, opening a named pipe blocks until both ends are connected. Opening for read and write in the same script without backgrounding one side causes deadlock.

Workaround: use exec 3<>pipe to open read-write on a single FD, or background the reader before the writer opens. Cleanup is manual — always pair mkfifo with trap 'rm -f pipe' EXIT.

Temporary Files

mktemp creates a seekable, randomly accessible file that persists until explicitly removed. It works across shell boundaries and survives reboots. The cost is physical disk I/O and mandatory cleanup. In world-writable directories like /tmp, always use mktemp -p /tmp myscript.XXXXXX or mktemp -t myscript.XXXXXX to avoid symlink races.

Concrete Validation: Benchmark Harness

The following test harness measures real, user, and sys time plus peak RSS for each method moving a 100 MB stream. Run it in a Bash 4.0+ environment (check with bash --version). No elevated permissions needed; it writes to /tmp.

#!/usr/bin/env bash
# benchmark-ipc.sh — compare process substitution, named pipe, temp file
set -euo pipefail

DATA_SIZE_MB=100
ITERATIONS=3
RESULTS_FILE=$(mktemp -t ipc-bench.XXXXXX)
trap 'rm -f "$RESULTS_FILE"' EXIT

echo "Method,Real(s),User(s),Sys(s),MaxRSS(KiB)" > "$RESULTS_FILE"

for method in procsub namedpipe tempfile; do
  for i in $(seq 1 "$ITERATIONS"); do
    case $method in
      procsub)
        # Process substitution: tee to two consumers via >()
        /usr/bin/time -f "procsub,%e,%U,%S,%M" \
          bash -c 'dd if=/dev/zero bs=1M count=100 2>/dev/null | tee >(cat >/dev/null) >(cat >/dev/null) >/dev/null' \
          2>>"$RESULTS_FILE"
        ;;
      namedpipe)
        # Named pipe: background reader, writer in foreground
        PIPE=$(mktemp -u -t ipcpipe.XXXXXX)
        mkfifo "$PIPE"
        trap 'rm -f "$PIPE"' EXIT
        /usr/bin/time -f "namedpipe,%e,%U,%S,%M" \
          bash -c "cat '$PIPE' >/dev/null & dd if=/dev/zero bs=1M count=100 2>/dev/null > '$PIPE'; wait" \
          2>>"$RESULTS_FILE"
        rm -f "$PIPE"
        trap - EXIT
        ;;
      tempfile)
        # Temporary file: write then read
        TMP=$(mktemp -t ipctemp.XXXXXX)
        trap 'rm -f "$TMP"' EXIT
        /usr/bin/time -f "tempfile,%e,%U,%S,%M" \
          bash -c "dd if=/dev/zero bs=1M count=100 2>/dev/null > '$TMP'; cat '$TMP' >/dev/null" \
          2>>"$RESULTS_FILE"
        rm -f "$TMP"
        trap - EXIT
        ;;
    esac
  done
done

column -t -s, "$RESULTS_FILE"

Run with bash benchmark-ipc.sh. Expected output: a CSV table showing timings. Process substitution typically shows lowest real time (no disk I/O). Named pipe adds overhead from filesystem entry creation. Temporary file shows highest sys time due to physical writes.

To observe subshell lifetimes and FD assignments, run any method under bash -x:

bash -x -c 'dd if=/dev/zero bs=1M count=10 | tee >(cat >/dev/null) >/dev/null'

Look for + tee /dev/fd/63 lines showing the assigned descriptor.

Decision Guide

1. Need POSIX portability? Use temporary files with mktemp and trap 'rm -f "$TMP"' EXIT.
2. Bash-only, high-throughput, no seek needed? Use process substitution. Zero setup, automatic cleanup, kernel-pipe speed.
3. Multiple independent readers or long-lived co-process? Use named pipe with exec 3<>pipe or backgrounded reader. Consider coproc (Bash-specific) for cleaner bidirectional pipes with explicit FD numbers and PID.
4. Need random access (seek, mmap)? Temporary file is the only option.

Limitations and Verification

• Kernel pipe buffer: All pipe-based methods (process substitution, named pipe) are limited by the kernel pipe buffer (typically 64 KiB). If the consumer is slower, the producer blocks — same as regular pipes.
• Signal propagation: Process substitution subshells run in a separate process group. trap 'kill 0' SIGINT SIGTERM in the main script forwards signals to the process group.
• Fallback behavior: On systems without /dev/fd, Bash creates a named pipe for process substitution. Verify with bash -c 'ls -la /dev/fd/ <(echo test)' — if it shows a pipe path instead of /dev/fd/N, the fallback is active.
• Cleanup verification: After any script run, check ls -la /tmp/ipc* 2>/dev/null || echo 'clean' to confirm no residual files.

Quick Verification Commands

Run these one-liners to confirm behavior on your target system:

# Confirm Bash version
bash --version

# Process substitution FD visibility
bash -c 'ls -la /dev/fd/ <(echo test)'

# Named pipe non-blocking open
mkfifo /tmp/testpipe; cat /tmp/testpipe & echo hello > /tmp/testpipe; wait; rm /tmp/testpipe

# Temporary file cleanup
trap 'rm -f "$TMP"' EXIT; TMP=$(mktemp); echo data > "$TMP"; cat "$TMP"

# Subshell PID difference
bash -c 'echo $BASHPID; <(sleep 1; echo $BASHPID >&2)'