Python List Comprehensions: A Practical Rule for When They Help and When They Hurt

The one-liner that got out of hand

Every Python codebase has one: a list comprehension that started life as a tidy [x * 2 for x in nums] and, six months later, has grown a nested loop, two conditionals, and a function call nobody remembers adding. It still works. Nobody dares touch it.

The thesis of this post is simple: a list comprehension is a readability tool first and a performance tool second. Use it when the transformation fits in your head at a glance; drop to a plain for loop the moment it doesn't. Everything else — speed, memory, side effects — follows from that decision.

What a comprehension actually is

A list comprehension like [name.upper() for name in users if name] is syntactic sugar. CPython compiles it into roughly the same bytecode as an explicit loop that calls list.append — you can confirm this yourself with the dis module:

import dis

def f(users):
    return [u.upper() for u in users if u]

dis.dis(f)

Run this in any Python 3 REPL (no special permissions needed) and you'll see a loop with a LIST_APPEND opcode — the append is a dedicated bytecode instruction rather than a method lookup, which is why comprehensions are often marginally faster than a hand-written loop. But "marginally" is the key word. On typical data sizes the difference is small enough that readability should win every tie. If you genuinely suspect a hot path, measure with timeit on your own data rather than assuming either form is faster.

One scoping detail worth knowing: in Python 3, the loop variable of a comprehension is local to the comprehension. It does not leak into the enclosing scope the way it did in Python 2. The surrounding variables you read are looked up normally, though — which matters for the side-effect trap covered below.

A worked example: the same logic, three ways

Suppose you're normalizing log records: keep only errors, extract the message, strip whitespace. Here's the comprehension version:

messages = [
    r["msg"].strip()
    for r in records
    if r.get("level") == "ERROR" and r.get("msg")
]

This is about the upper limit of what I'd accept as a comprehension: one expression, one loop, a compound filter. The loop version is more verbose but easier to step through in a debugger:

messages = []
for r in records:
    if r.get("level") == "ERROR" and r.get("msg"):
        messages.append(r["msg"].strip())

And here's the version that should never have been a comprehension — the same shape people reach for when the data is nested:

# Hard to read, hard to debug, hard to extend
flat = [m.strip() for batch in batches for r in batch
        if r.get("level") == "ERROR" for m in [r.get("msg") or ""] if m]

Multiple for clauses read left-to-right as nested loops, which is backwards from how most people scan code. Once you need a second for or a conditional expression inside the value slot, write the loop. The loop also gives you natural places for logging, breakpoints, and try/except around individual items.

The two real traps: memory and side effects

Eager evaluation. A list comprehension builds the entire result in memory before you can use any of it. For a few thousand records, fine. For streaming a multi-gigabyte file, it's a real problem. The fix is a generator expression — same syntax, parentheses instead of brackets — which produces items lazily:

total = sum(len(line) for line in open("app.log"))  # lazy, constant memory

Note you can drop the extra parentheses when the generator is the only argument to a function. The trade-off: generators are single-use and don't support indexing or len(), so if you need to iterate the result twice, materialize a list.

Side effects. Because a comprehension can read any variable from the enclosing scope, it's tempting to mutate shared state inside one — appending to a second list, incrementing a counter, calling a function that writes somewhere. This hides the mutation in a construct readers assume is a pure transformation. If your comprehension body does anything besides computing the output value, that's a loop wearing a costume.

A decision rule you can apply in review

When reviewing (or writing) a comprehension, ask three questions:

1. Is there exactly one for clause and at most one or two simple conditions?
2. Is the value expression a single, side-effect-free operation?
3. Will the whole result fit comfortably in memory, and do I need it all at once?

Three yeses: keep the comprehension. A no on the third question: switch to a generator expression. A no on either of the first two: write the explicit loop and enjoy your future debugger sessions.

Closing

List comprehensions are one of Python's best features precisely because they're constrained — the moment you fight that constraint, the feature stops paying for itself. Pick one comprehension in your current codebase that fails the checklist above, rewrite it as a loop, and see whether the diff makes the intent clearer. That ten-minute exercise teaches the boundary better than any style guide.