Transducer pipeline or sequential map for low‑traffic Clojure services: which yields lower allocation overhead?

Answer

For a Clojure 1.11+ service under ~10 requests per second, a transducer pipeline generally yields lower per-item allocation than sequential map with doall, but the absolute saving at very low traffic is small and dominated by request handling overhead. The transducer advantage persists even below 1 req/s when the transducer is reused across requests; for one-off tiny transformations the simpler map+doall is acceptable for readability with negligible GC impact.

Confirmed facts

• Transducers compose reducing functions and process items with a single reduce pass. They avoid intermediate lazy seq objects that map creates.
• Sequential map is lazy. doall forces realization, allocating seq nodes per element and an intermediate collection.
• Transducers were stabilized in Clojure 1.7 and are stable in 1.11+. Ordering of stages is preserved in the composed reducing function.
• Pipeline ordering affects both approaches. Early filtering or mapping reduces downstream work for both.

Likely explanation

• For very small collections and request rates below ~10/s, the absolute allocation difference is typically small and often dominated by HTTP, I/O and server overhead rather than transformation.
• Transducer setup cost is amortized over many items. With tiny inputs the benefit can be comparable to map+doall.
• With transducers, placing cheap filters first reduces allocations in later stages without creating intermediate seqs. With map+doall, early filtering still requires intermediate seq nodes before realization.
• Collection strategy does not reduce thread-pool wake-ups or idle responsiveness. Those are governed by server and executor configuration, not by map vs transducer.
• core.async channels add bookkeeping and buffer allocations that can outweigh transducer savings for low-traffic services.

Steps needed for this case

1. Reuse a transducer. Define it once with comp and do not recreate it per request.
2. Order stages with cheap filters first, then maps, then expensive operations.
3. Measure on representative input sizes and request rate. Compare a shared transducer used with into or transduce against map+doall using JVM GC logs and allocation rate.
4. Confirm Clojure version is 1.11 or later and that the transducer is not created per request, as per-request creation adds allocation overhead.

Missing diagnostic that changes the recommendation: typical collection size per request and whether the transformation is reused as a shared transducer instance. Reuse strongly favors transducers; single-use tiny vectors favor simplicity.