Diagnosing and Fixing Memory Leaks Caused by Lazy Sequence Head Retention in Clojure

The Problem: Progressive Heap Growth and GC Pressure

A common failure mode in Clojure JVM applications is a slow, steady increase in heap usage that eventually leads to frequent Garbage Collection (GC) pauses and OutOfMemoryError. This often occurs after introducing data processing pipelines using map, filter, or remove, especially when the results are stored in long-lived state like atoms, caches, or global vars.

The root cause is usually head retention. In Clojure, a lazy sequence is a promise of a computation. If you hold a reference to the start (the head) of a lazy sequence while iterating through it, the JVM cannot garbage collect the elements you have already processed because they are still reachable via that head reference.

Diagnostic Matrix: Identifying the Cause

Symptom	Likely Cause	Diagnostic Indicator
Steady heap growth under constant load	Head retention in long-lived collection	Heap dump shows high count of clojure.lang.LazySeq objects
Latency spikes during processing	Repeated partial realization	High allocation rate of intermediate LazySeq objects
Rapid OOM with large inputs	Unbounded source sequence	Stack trace points to a reduce or doall on an infinite seq
Memory not reclaimed after request	Closure capture of binding context	Retaining path from a long-lived function/callback to a large local var

Ordered Diagnostic Checks

1. Analyze Heap Dumps: Use a tool like VisualVM or Eclipse MAT. Search for clojure.lang.LazySeq. If these dominate the heap, examine the "Path to GC Root." Look for references held by clojure.lang.Atom, clojure.lang.Ref, or static vars.
2. Audit API Boundaries: Review functions that return the result of map or filter. Check if the caller stores these results in a cache or a stateful object without first realizing them.
3. Scan for Global State: Search the codebase for def or defonce statements that hold sequences. Check if these sequences are updated via pipelines that might be retaining the head of the previous version.
4. Isolate and Reproduce: Create a minimal REPL test case using a smaller dataset. Observe memory behavior before and after forcing realization at the suspected boundary.

Fixes Based on Findings

Finding: Head is retained in a long-lived reference

If you are storing a sequence in an atom or cache, you must realize the sequence into a strict collection (like a vector) to break the link to the lazy computation chain.

;; RISK: Retains head if 'processed-data' is held in an atom
(def state (atom (map transform-fn large-source)))

;; FIX: Realize into a vector before storing
(swap! state (fn [_] (vec (map transform-fn large-source))))

Use vec or into [] ... to force immediate realization. Avoid doall if you intend to store the result, as doall returns a lazy sequence that is already realized but still carries the overhead of the lazy structure.

Finding: Lazy sequences returned to callers

When a library or internal service returns a lazy sequence, the ownership of realization is ambiguous. Establish a contract: either the producer realizes the data, or the consumer must realize it immediately upon receipt.

Recommended Pattern: Realize at the system boundary (e.g., just before sending a JSON response in a Ring handler).

Finding: Unbounded source sequences

If the source sequence is infinite or potentially massive, add explicit limits using take to prevent the JVM from attempting to realize the entire set into memory.

;; RISK: May realize millions of items
(filter predicate source-seq)

;; FIX: Apply a hard limit
(take 1000 (filter predicate source-seq))

Finding: High allocation during collection building

If you are building a large collection in a loop, use transients. Transients provide a mutable version of a collection for efficient construction, which is then "frozen" into an immutable one.

Caution: Transients must never escape the local scope where they are created; leaking a transient can lead to non-deterministic state corruption.

Verification and Testing

To verify the fix, run the following check in a REPL with a profiler attached:

1. Create a lazy pipeline: (def s (map inc (range 1000000))).
2. Store the head: (def head s).
3. Consume the sequence: (doseq [x s] (do nothing)).
4. Observe that memory remains high because head prevents GC.
5. Repeat, but replace the storage with (def head (vec s)) and verify that the original lazy chain is eligible for collection.

Escalation Criteria

Escalate the issue to a platform engineer or JVM specialist if:

• Heap dumps show dominator roots in the REPL or dynamic vars that cannot be modified.
• Memory growth persists despite all boundary realizations and a full code audit of retained heads.
• The leak is suspected to be in a third-party Java library or related to JVM classloader leaks.
• Native memory (off-heap) pressure is observed despite a stable JVM heap.