Two-File Lookups in Awk Without Extensions: The FNR == NR Pattern

Need to enrich a stream of records with a small reference table, and you want the same one-liner to run on any POSIX box without installing gawk extensions. The portable way is awk's FNR == NR idiom for a two-file lookup.

FNR is the record number within the current file. It resets to 1 at the start of each new file. NR is the record number across all files processed so far. While awk reads the first file on the command line, FNR == NR is true. As soon as the second file starts, FNR resets but NR does not, so the test becomes false. That difference gives you a clean first-pass / second-pass split without ARGIND or other non-portable features.

How the split builds a lookup

The conventional pattern is:

awk 'FNR == NR { map[$1] = $2; next } { print $0, map[$key] }' lookup.txt data.txt

Run this in a shell where you have read permission for both files. The first block executes only while reading lookup.txt. It stores key-value pairs in an associative array map, then next skips the default print so the lookup file itself is never emitted. When data.txt is read, the array is already in memory and can be used to enrich each record.

Replace $1, $2 and $key with the fields that actually identify a row in your files. Set a field separator explicitly with -F or FS if your files are not whitespace-delimited, otherwise mismatched delimiters cause silent mismatches.

Concrete enrichment example

Suppose codes.txt contains code and description pairs, one per line, and events.log contains event_id user_id code. You want to append the description to each event.

awk -F, 'FNR == NR { desc[$1] = $2; next } { print $0, desc[$3] }' codes.txt events.log

Here lookup.txt is codes.txt with -F, and the join key in the data file is $3. The command requires read access to both files and no write permission. A practical check is to run the same command with mawk, nawk and gawk if available and confirm identical behavior, because the idiom is specified by POSIX and does not rely on gawk extensions.

gawk provides ARGIND to identify the current file index, which can make intent clearer, but ARGIND is not portable to mawk or nawk. Sticking to FNR == NR keeps the script runnable everywhere.

Trade-offs and edge cases to verify

The entire first file is held in memory as an associative array. This is fine for reference tables that fit in RAM and is not a streaming join for multi-gigabyte lookup tables. Very large lookups can exhaust memory.

If the first file contains duplicate keys, awk silently keeps the last value encountered. No warning is emitted. Test with intentional duplicates to confirm which value wins for your data.

An empty first file is a subtle failure mode: FNR resets to 1 on the second file, so FNR == NR becomes true again and the second file would be treated as the lookup. Guard the logic with an explicit file name check or ensure the lookup is non-empty before relying on the pattern.

Field separators must be consistent. Explicitly set -F for both files or normalize them beforehand, otherwise key matching fails silently.

A useful verification step is to create small test files with known keys, run the command, and inspect whether matched rows show a mapped value and unmatched rows show an empty field. Also test the empty lookup and duplicate key cases to observe the documented behavior.