awk length() and substr() Limits Under Byte and Character Locales

Answer

For a user‑facing awk pipeline that may contain non‑ASCII text, character‑correct counting should take precedence. Using length(), substr(), etc. under a UTF‑8 locale gives results that match the number of characters users see, preventing mis‑alignment or truncation when the output is displayed or read by a screen reader. Byte‑exact reproducibility (LC_ALL=C) is only preferable when the output contract is defined in bytes and the data are known to be ASCII‑only (e.g., machine‑generated logs or protocols).

Display‑width formatting

Awk’s length() counts characters, not display width. Combining marks, zero‑width characters, and East Asian wide characters cause the visual column width to differ from the character count. Therefore, delegate alignment or padding to a tool with an explicit width model (e.g., wcwidth/wcswidth or a dedicated formatting utility) rather than relying on length() alone.

Runtime detection of multibyte support

You can test whether the installed awk honors multibyte locales with a simple one‑liner:

# Test string: UTF‑8 'é' (two bytes, one character)
printf '%s\n' "é" | awk -v LC_ALL=C '{print length($0)}'   # byte count under C
printf '%s\n' "é" | awk '{print length($0)}'               # count under current locale

If the two results differ (e.g., 2 vs. 1), the awk binary respects multibyte locales. You can also inspect awk --version or the implementation’s manual for compile‑time multibyte flags.

Missing diagnostic detail

The recommendation hinges on whether the pipeline’s output is intended for human consumption or for further machine processing. If you can confirm that the output is strictly ASCII‑only and machine‑consumed, byte‑exact reproducibility (LC_ALL=C) may be the better choice; otherwise, favor character‑correct counting.