Speeding Up Vector Construction in Clojure with Transients

Concrete Problem: Building Huge Vectors is Slow

In many data-processing pipelines you end up constructing a vector that holds millions of elements. The idiomatic way in Clojure is to start with an empty vector and conj each new item:

(reduce conj [] (range 10_000_000))

Because Clojure collections are immutable, each conj creates a new vector that shares most of its structure with the previous one. The cost of that sharing, plus the overhead of allocating a fresh persistent object for every step, adds up quickly. On a 10-million element build, you can see a noticeable lag and a spike in garbage-collector activity.

Thesis: Use Transients for Fast, In-Place Construction

Transients are a built-in Clojure feature that gives you a mutable, low-overhead representation of a persistent collection. You can mutate it inside a single thread, then convert it back to a persistent form with persistent!. The resulting vector is identical to one built purely with conj, but the construction phase is much faster and generates fewer garbage objects.

What Are Transients?

• Mutable wrapper around a persistent collection.
• Safe only within the thread that created it.
• Converted back with persistent! before sharing.
• Available on vectors, lists, maps, and sets.

Using Transients for Vector Construction

Below is a minimal pattern you can drop into any Clojure project. It demonstrates building a vector of numbers from 0 to n-1 in two ways: the classic persistent approach and the transient approach.

(defn build-persistent [n]
  (reduce conj [] (range n)))

(defn build-transient [n]
  (let [t (transient [])]
    (dotimes [i n]
      (conj! t i))
    (persistent! t)))

Key points:

• transient [] creates a mutable vector.
• conj! mutates the transient in place.
• After the loop, persistent! freezes the structure so it can be safely shared.
• All operations run on the same thread; no synchronization is needed.

Benchmarking the Difference

Using criterium you can compare the two approaches. Run this in a REPL with criterium on your classpath (no special permissions needed):

(require '[criterium.core :refer [bench]])

(bench (build-persistent 10_000_000))
(bench (build-transient 10_000_000))

Reported results for this kind of comparison commonly show the transient version several times faster (up to roughly 5x on large builds), with substantially less object churn. Verify on your own hardware and JDK, since absolute numbers vary. You can also enable GC logging (e.g., -Xlog:gc on modern JVMs) to confirm the reduction in allocations: the persistent build creates millions of intermediate nodes, while the transient build mutates in place and allocates mainly the final structure.

Trade-offs & Limitations

• Thread safety: Transients are not thread-safe. If you accidentally share a transient across threads you'll corrupt data. Always call persistent! before passing the result out of the thread that created it.
• Forgotten conversion: If you forget to call persistent!, you'll expose a mutable object to the rest of your program. This can lead to subtle bugs that are hard to trace.
• Small collections: For vectors with fewer than a few hundred elements the overhead of creating a transient and converting it back can outweigh the benefit. Use a size heuristic if you want to be safe.
• API familiarity: The transient API (e.g., conj!, assoc!, pop!) is slightly different from its persistent counterpart. Make sure your team is comfortable with the bang-suffix naming convention.

Practical Take-away: A Ready-to-Use Pattern

When you need to build a large collection, wrap the construction in a transient block and convert it back only once:

(defn fast-build [source]
  (let [t (transient [])]
    (doseq [x source]
      (conj! t x))
    (persistent! t)))

Use this pattern for:

• Bulk imports from files or databases.
• Pre-processing steps that accumulate a large result.
• Any scenario where you build a collection in a single thread and then expose it to the rest of the system.

Remember to add a guard or documentation comment if your function is part of a public API: "This function uses transients internally; it returns a persistent vector." That keeps the contract clear and protects callers from accidental misuse.

In short, transients give you the speed of mutable data structures while preserving the purity guarantees of Clojure's persistent collections—provided you respect the single-thread rule and convert before sharing.