When Awk’s Associative Arrays Hit Their Limits: A Practical Guide to Single‑Pass Aggregation

The problem: counting unique values in a huge log

You have a 10 GB web‑access log and need to know how many distinct IP addresses appear each day. Writing a multi‑pass script with sort | uniq -c would require sorting the whole file, which is slow and needs extra disk space. You wonder whether a single‑pass awk script can do the job without blowing up memory.

How awk stores data

Awk’s arrays are hash tables that keep only string keys. Numeric indices are silently converted to strings, so arr[1] and arr["1"] refer to the same slot. If you need a composite key (for example, IP,date), you must insert a delimiter yourself. The built‑in variable SUBSEP (default "\034", the file separator) is safe because it rarely appears in data:

# count per‑IP per‑day
awk -F' ' '{ip=$1; date=strftime("%Y-%m-%d",$4); key=ip SUBSEP date; counts[key]++} END {for (k in counts) print k, counts[k]}' access.log

The array grows with the number of distinct keys, not with the total number of records. A file with 10 GB of lines but only 50 000 unique IPs will keep a modest array; 10 million unique session IDs could exhaust RAM.

Watching memory usage

You can check how many keys awk has stored at the end of a run:

awk 'END {print "distinct keys:", length(counts)}' access.log

If you need a rough estimate while the script is running, pipe the output through /usr/bin/time -v and look for the “Maximum resident set size” field.

Worked example: daily unique‑IP count with CSV‑safe field splitting

Suppose the log is actually a CSV where the first field is the IP, the second is the timestamp, and other fields may contain commas inside quoted strings. GNU awk 4.0+ provides FPAT to parse RFC‑4180 CSV without writing your own quote handling:

# gawk 4.0+ only
LC_ALL=C gawk -v FPAT='([^,]*)|(\"[^"]+\")' '
{
    ip = $1
    # timestamp is in $2, format: 2026/10/10 14:32:05
    split($2, ts, " ")
    date = ts[1]
    key = ip SUBSEP date
    uniq[key]++
}
END {
    PROCINFO["sorted_in"] = "@ind_num_asc"   # optional ordering
    for (k in uniq) {
        split(k, parts, SUBSEP)
        print parts[1], parts[2], uniq[k]
    }
}' traffic.csv

This script makes a single pass, builds the associative array uniq, and prints each IP‑date pair with its count. The LC_ALL=C setting ensures that number formatting and string comparisons are deterministic across locales.

Trade‑offs and limitations

• Memory ceiling: The array lives in RAM for the whole duration of the awk process. If the distinct‑key count approaches the available memory (e.g., > 1 million keys on a modest machine), consider switching to a tool that can spill to disk, such as sort | uniq -c, a lightweight database (SQLite), or a streaming framework like Apache Flink.
• No persistence: Arrays vanish when the script ends. For checkpointing you would need to write intermediate results to a temporary file and reload them in a second pass.
• Locale sensitivity: Awk uses the current locale for numeric formatting and string ordering. A German locale, for example, prints 4,0 instead of 4.0. Setting LC_ALL=C removes this variability.
• Feature availability: FPAT, PROCINFO["sorted_in"], and nextfile are GNU‑awk or mawk extensions. Verify with awk --version before relying on them in portable scripts.

Actionable closing

Before committing to an awk‑only solution, run a quick sampler on a subset of your data:

# sample 1 % of the file and count distinct keys
awk -F' ' '{ip=$1; date=strftime("%Y-%m-%d",$4); key=ip SUBSEP date; seen[key]++} END {print "distinct keys in sample:", length(seen)}' <(shuf -n $(($(wc -l < access.log)/100)) access.log)

If the sampled distinct‑key count multiplied by the inverse sample size stays comfortably below your RAM limit, the single‑pass awk approach is likely safe. Otherwise, plan a hybrid strategy: use awk for the initial filter‑transform step, then feed the reduced stream into a sort‑based or database aggregation step.

---

Awk remains a sharp knife for “filter‑transform‑aggregate” workloads under a few hundred thousand distinct keys. Knowing where its edge dulls helps you choose the right tool before the job runs out of memory.