The Unix Pipe: How One Character Turns Small Tools into a Data Engine

You need the ten largest log lines from a 4 GB file, deduplicated, right now. You could write a Python script. Or you could type one line of shell and get the answer before the script's imports finish. The difference is the pipe — the humble | — and understanding what it actually does explains both why it feels like magic and where it quietly bites you.

The thesis: a pipe is not a file, not a variable, and not a convenience syntax. It is a kernel-managed, unidirectional byte channel with a fixed-size buffer, and every behavior you love (no temp files, parallel execution) or hate (lost exit codes, mysterious stalls) follows from that one fact.

What the shell actually does when you type |

When you run sort access.log | uniq -c | sort -nr | head, the shell does not run sort, wait, save the output, and feed it onward. For each | it calls pipe() to get a pair of file descriptors — one read end, one write end — then fork()s a child per command. In each child, dup2() rewires descriptor 1 (stdout) or 0 (stdin) onto the pipe end, and exec() replaces the child with the real binary. The binaries never know. sort thinks it is writing to a terminal; uniq thinks it is reading a file.

Two consequences matter. First, all stages run concurrently. There is no "step 1 finishes, step 2 starts." Second, nothing touches disk. Data lives in a kernel buffer (commonly 64 KB on modern Linux; check man 7 pipe for your system) and flows straight from one process's memory to another's.

Backpressure: the pipe's hidden synchronization

The buffer is finite, and that is a feature. If the writer outpaces the reader, the writer's write() blocks until the reader drains some bytes. If the reader is ahead, its read() blocks until data arrives. This is backpressure — automatic flow control with zero code from you.

It also means a pipeline is only as fast as its slowest stage. If grep in the middle is CPU-bound, everything upstream politely parks itself. You can watch this: run a heavy pipeline and check ps or top — you will see every stage alive simultaneously, most of them sleeping in a blocking read or write.

A worked example: top offenders in a log

Run this in any POSIX shell, as an unprivileged user, against a web server access log in the common format:

awk '{print $1}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -nr | head -10

Each stage does one thing: awk extracts the client IP field, the first sort groups identical IPs adjacently (a requirement for uniq), uniq -c counts them, sort -nr sorts numerically in reverse, and head -10 keeps the top ten. To verify it works, sanity-check one line: pick the top IP and run grep -c "^1.2.3.4 " /var/log/nginx/access.log — the count should match what uniq -c reported.

There is a bonus behavior here: when head exits after ten lines, the next write to its pipe fails and upstream sort receives SIGPIPE and dies. The pipeline cleans itself up. No orphan processes grinding through data nobody wants.

The trade-off: exit codes lie by default

Here is the sharp edge. A pipeline's exit status is, by POSIX default, the exit status of the last command only. In curl https://example.com/data | jq ., if curl fails with a 404 and writes an HTML error page, jq may still exit 0 on some inputs — and your script sails on with garbage. In bash, fix this with set -o pipefail at the top of scripts, which makes the pipeline return the rightmost non-zero status. Verify with: false | true; echo $? prints 1 with pipefail, 0 without.

Other limits worth knowing: pipes are unidirectional (for two-way chatter you need a FIFO via mkfifo or a socket), they carry raw bytes with no message boundaries (fine for line-oriented text, awkward for structured records), and they only work between related processes on one machine — no network, no persistence.

Where to go from here

Next time you reach for a throwaway script, try composing it from filters first. Keep each stage line-oriented and single-purpose, add set -o pipefail in anything scripted, and read man 7 pipe once — it is short and explains the buffer semantics that govern everything above. The pipe is fifty years old and still the cheapest concurrency primitive you will ever use.