Awk Associative Arrays: Aggregation Without the Sort Pipeline

The log-summarizing problem everyone solves the slow way

You have a web server access log and a simple question: how many requests did each IP make? The reflex answer is cut -d' ' -f1 access.log | sort | uniq -c | sort -rn. It works, but it sorts the entire stream before counting, which costs O(n log n) time and a full intermediate copy of the data.

Awk answers the same question in one pass, with memory proportional to the number of distinct keys rather than the size of the file:

awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log

The mechanism behind this — awk's associative arrays — is the single most useful feature in the language, and it behaves identically in POSIX awk, gawk, and mawk. Understanding three of its quirks will save you from subtle bugs.

Why count[$1]++ works with no setup

Awk arrays spring into existence on first use. There is no declaration, and an element you have never touched reads as zero in a numeric context (or the empty string in a string context). That is why count[$1]++ is safe on the first occurrence of an IP: the unset element reads as 0, then increments to 1.

Indices are always strings. If a field looks numeric, awk converts it, but the key is stored as a string. This has one surprising consequence: 01 and 1 are different keys, because they are different strings. If your input mixes zero-padded and plain numbers, force numeric normalization with count[$1 + 0]++.

The membership-test trap: in versus ==

Here is the quirk that bites people on large inputs. Merely referencing an element creates it:

# BAD: creates arr[k] as a side effect of testing it
if (arr[k] == "") { ... }

# GOOD: tests membership without creating anything
if (k in arr) { ... }

The first form auto-vivifies the key — it now exists in the array with an empty value. On a deduplication job over hundreds of millions of lines, that is the difference between storing only the keys you care about and storing every key you ever glanced at. It also changes later behavior: after the bad test, (k in arr) returns true even though you never inserted anything. Use the in operator for membership tests, always.

A worked example: bytes per URL

Counting is the simple case. Summing a column grouped by another column is the same pattern with addition. Given a log where field 7 is the request path and field 10 is the response size:

awk '
  $10 ~ /^[0-9]+$/ { bytes[$7] += $10; hits[$7]++ }
  END {
    for (url in bytes)
      printf "%12d %6d %s\n", bytes[url], hits[url], url
  }
' access.log

Run this from a shell against the log file; it needs only read permission on the file. The regex guard skips malformed lines (some log formats write - for missing sizes). Two arrays keyed by the same field stay in lockstep because both are updated in the same block.

Validate the result against the classic pipeline on a small sample:

head -n 10000 access.log | awk '{print $7}' | sort | uniq -c | wc -l
head -n 10000 access.log | awk '{h[$7]++} END {print length(h)}'  # length(array) is a gawk extension

Both counts of distinct URLs should match. For a strictly portable size check, iterate instead: END { for (u in h) n++; print n }.

Ordering and cleanup

Two final mechanics. First, for (k in arr) iterates in an unspecified order — POSIX guarantees you see every key, not the sequence. If you need sorted output, pipe into sort after awk, or in gawk set PROCINFO["sorted_in"] = "@val_num_desc" before the loop (a gawk-only extension).

Second, delete arr[key] removes one element and delete arr clears the whole array; both are POSIX. Deleting as you go is useful in streaming jobs that flush aggregates per time window, keeping memory bounded.

The trade-off: everything lives in memory

Awk's one-pass aggregation holds every distinct key in RAM. For access logs, request IDs, or typical cardinality — thousands to a few million keys — that is fine and much faster than sorting. At hundreds of millions of distinct keys, memory blows up and a sort-based pipeline (which spills to disk gracefully) or a real database wins. The honest rule: awk for one-liners and moderate cardinality, external sort or a database when the key space is the problem.

Also remember the portability line: everything shown here except length(array) and PROCINFO["sorted_in"] is POSIX and behaves the same in gawk and mawk. Check awk --version or man awk on the target system before relying on extensions.

Try it on your next log

Next time you reach for sort | uniq -c, ask whether you actually need the sort. If the answer is "I just want counts," the awk version is shorter, faster, and one process instead of three. Start with { count[$1]++ }, test membership with in, and pipe to sort only when the output order matters.