Limits of bash `read -r -d ''` NUL byte truncation in piped input

Answer

When Bash reads from any descriptor (pipe, regular file, or the shell’s own stdin) with read -r -d '', it stops at the first NUL byte and discards that byte. The resulting shell variable contains everything before that NUL. This behavior is deterministic, the same for pipes and files, and is defined by POSIX, not by the underlying read() system call. The -r flag only turns off backslash interpretation; it does not affect NUL handling. To preserve binary data, avoid storing it in a shell variable; instead write it to a file or pipe it directly into a binary‑safe utility.

Why It Happens

The Bash read builtin parses input as a text string. It keeps reading bytes until it encounters the delimiter specified by -d. If -d '' is used, the delimiter is the NUL byte. As soon as that byte is seen, the builtin stops and does not include it in the variable. The C library’s read() syscall returns raw bytes, but Bash interprets the stream according to its own rules.

POSIX Specification

POSIX §2.6.2.1 describes the read builtin: “If the -d option is supplied, the first instance of the delimiter character is discarded and the string read is terminated before that character.” When the delimiter is NUL, the string is terminated at the first NUL. This specification applies to all POSIX‑conforming shells, so the truncation is not an implementation quirk.

Effect of -r

The -r flag disables backslash escape processing. It has no bearing on how NUL bytes are treated; the variable will never contain a NUL regardless of -r.

Deterministic Truncation Point

The truncation point is always the first NUL byte encountered in the input stream. Whether the stream comes from a pipe or a file does not change this. Example verification:

# Create a file with a NUL in the middle
printf 'abc\0def' > test.bin

# Read from a pipe
cat test.bin | read -r -d '' var; echo "[$var]"
# → [abc]

# Read directly from a file
read -r -d '' var < test.bin; echo "[$var]"
# → [abc]

Portable Workarounds

• Write the binary data to a temporary file: dd if=/dev/stdin bs=1 of=tmpfile or cat >tmpfile.
• Encode the data before storing it in a variable: base64 <tmpfile or xxd -p <tmpfile.
• Pipe the data directly into a binary‑safe utility (e.g., openssl enc, gzip, sha256sum) without ever placing it in a shell variable.

When you need to process binary data in a shell pipeline, replace read -r -d '' with a construct that preserves the entire stream. For example:

cat input.bin | dd bs=1 of=output.bin
# or simply
cat input.bin > output.bin

After the data is safely stored in a file, any downstream processing can be performed with tools that understand binary input.

Summary

• read -r -d '' always truncates at the first NUL, regardless of the input source. • POSIX mandates this behavior; it is not a library implementation detail. • -r only disables backslash escapes. • Use file I/O or binary‑safe utilities for binary data; avoid shell variables when NULs may appear.