Counting Unique IP Addresses in Logs with awk Associative Arrays

Problem: tallying IPs without extra sorting or temp files

When you need to know how many times each IP address appears in a web server log, the usual pipeline cut -d' ' -f1 access.log | sort | uniq -c works but forces the shell to sort the entire list first. For large logs this creates noticeable CPU and I/O overhead and requires temporary disk space for the sort step.

Thesis: awk’s built‑in associative arrays count in a single pass

awk automatically creates an associative array when you reference a key that has not been seen before. By incrementing ip[$1] for each line, you build a frequency table while reading the file only once. The END block then prints the collected counts.

Worked example

Assume a tiny test log access.log with the following lines:

192.168.1.10 - - [13/Oct/2026:12:00:01 +0000] "GET /index.html HTTP/1.1" 200 512
10.0.0.5 - - [13/Oct/2026:12:00:02 +0000] "GET /style.css HTTP/1.1" 200 1024
192.168.1.10 - - [13/Oct/2026:12:00:03 +0000] "GET /script.js HTTP/1.1" 200 768
10.0.0.5 - - [13/Oct/2026:12:00:04 +0000] "GET /image.png HTTP/1.1" 200 2048
192.168.1.10 - - [13/Oct/2026:12:00:05 +0000] "GET /favicon.ico HTTP/1.1" 200 123

The awk one‑liner:

awk '{ip[$1]++} END {for (a in ip) print ip[a], a}' access.log

produces:

3 192.168.1.10
2 10.0.0.5

Each line shows the count followed by the IP address. To order by frequency you can pipe the output through sort -nr.

Trade‑off: memory usage and numeric‑key caution

The associative array holds one entry per distinct key. If the log contains hundreds of thousands of unique IPs, the resident memory grows proportionally. On a system with limited RAM this may cause swapping or out‑of‑memory conditions.

Additionally, awk treats array keys as strings but will convert them to numbers when they look numeric. For example, keys 001 and 1 are considered identical, which can collapse distinct values unintentionally. When your field might contain leading zeros (e.g., VLAN IDs or octet‑padded numbers), prefix the key with a non‑digit to keep it string‑only: ip["_".$1]++.

Verification and portability

1. Correctness: run the awk command on a small test file and compare its output to sort access.log | uniq -c. The counts should match.

2. Memory observation: generate a larger log (e.g., 1 M lines with 100 k unique IPs) and run /usr/bin/time -v awk '{ip[$1]++} END {for (a in ip) print ip[a], a}' big.log. The “Maximum resident set size” field shows the memory footprint.

3. Portability: test the same one‑liner with awk (POSIX), gawk (GNU), and nawk or bsdawk (BSD). The output should be identical across implementations.

Actionable closing

For log‑analysis tasks that require a simple frequency tally, reach for awk’s associative arrays first. They give you a single‑pass, dependency‑free solution that is easy to embed in scripts or run directly from the shell. Just keep an eye on the cardinality of your key space and guard against unintended numeric conversion when needed.