Counting Things in Awk: Associative Arrays as Your First Aggregation Tool

You have a log file with a few million lines, and someone asks: how many requests did each endpoint get? You could load it into Python, spin up a dataframe, or push it into a database. But the file is already on the machine, and awk is already installed. With one pattern — an associative array keyed on a field — you get the answer in a single pass.

This post is about that pattern: using awk's associative arrays for frequency counting and grouping, why it works so well, and where it stops working.

The core pattern: count[key]++

An associative array in awk maps arbitrary strings to values. You never declare it, never size it, never seed it. The first time you write count[$1]++, awk creates the entry, treats the unset value as zero, and increments it. That one expression is a complete frequency map.

Because keys are just strings, there is no schema to define up front. Whatever shows up in field one becomes a key. This is what makes awk so effective for exploratory work on logs and CSV-ish files: you point it at a column and immediately get a histogram.

The second half of the pattern is the END block. It runs once, after every input line has been processed, which makes it the natural place to iterate over what you accumulated:

awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' input.txt

Run this in a shell, in the directory containing input.txt. No special permissions are needed — awk only reads the file. Replace $1 with whichever field you want to group by, and set FS if the file is not whitespace-delimited (for example, awk -F, '...' for CSV). The output is one line per unique key with its count. Nothing is written back; the file is untouched.

A worked example: requests per endpoint

Say your access log looks roughly like this, with the request path in field seven:

10.0.0.1 - - [13/Oct/2026:06:59:01 +0000] "GET /api/users HTTP/1.1" 200 432
10.0.0.2 - - [13/Oct/2026:06:59:02 +0000] "POST /api/login HTTP/1.1" 401 12

The request line itself contains spaces, so field seven is the path. A per-endpoint count is:

awk '{ count[$7]++ } END { for (path in count) print count[path], path }' access.log

Printing the count first is deliberate: it lets you pipe straight into sort -rn | head -20 to get the top twenty endpoints numerically, most frequent first. That pipeline — awk to aggregate, sort to rank — covers a surprising share of ad-hoc log analysis.

To check the result is sane, compare against a ground truth: wc -l access.log should equal the sum of all counts. A quick way to sum them:

awk '{ count[$7]++ } END { for (p in count) t += count[p]; print t }' access.log

If the total is short, some lines did not match the expected field layout (a common cause: requests with empty paths or malformed log lines shifting fields).

Grouping by more than one thing

Awk supports multi-dimensional-looking keys through a comma syntax: count[$7, $9]++ groups by path and HTTP status together. Internally awk concatenates the fields with a separator character (SUBSEP, default "\034"), so $7, $9 is really one composite string key.

To split it back apart when printing, use split():

awk '{ count[$7, $9]++ }
     END { for (k in count) { split(k, a, SUBSEP); print a[1], a[2], count[k] } }' access.log

One caution here: because the composite key is string concatenation, implicit string/number conversion can produce collisions you did not expect. A status read as the number 200 and the string "200" usually behave the same, but fields like "01" versus 1 can merge or diverge depending on how awk coerces them. If a key's exact text matters, force string treatment by concatenating with an empty string: key = $1 "".

The trade-offs you should know before relying on it

Memory scales with unique keys, not lines. Awk streams input line by line, so a billion-line file is fine — but every distinct key lives in memory until the program ends. Counting endpoints (thousands of keys) is trivial. Counting unique IP addresses or user IDs across a huge dataset (tens of millions of keys) can exhaust RAM and kill the process. If you suspect a high-cardinality key, check first with awk '{print $1}' file | sort -u | wc -l — though on a truly large file, that sort is itself expensive, so sampling with head is a cheaper sanity check. For genuinely high-cardinality aggregation, a sort-based pipeline (sort | uniq -c, which can spill to disk) or a real database is the better tool.

Output order is unspecified. The for (k in count) loop returns keys in whatever order the implementation's hash table yields. POSIX awk makes no ordering promise, and gawk, mawk, and nawk will typically print the same data in different orders. If order matters, pipe through sort — do not reach for gawk's asorti() unless you know gawk is available, since it is a GNU extension and will fail on a minimal awk. Check which awk you have with awk --version (gawk prints a version banner; others may error on the flag, which is itself a hint).

When to reach for this

The rule of thumb: if your aggregation fits in memory as a key set and you can express it as "increment something per line," awk's associative arrays are the fastest path from question to answer — no imports, no setup, no schema. The moment you need sorted output inside the script, joins across files with complex logic, or grouping over tens of millions of distinct keys, graduate to a heavier tool. But for the daily "how many of each thing are in this file," { count[$N]++ } plus an END loop is hard to beat.