Diagnosing and Fixing Clojure LazySeq Head Retention

The Problem: Memory Growth via Head Retention

In Clojure, lazy sequences cache their realized elements. If your code maintains a reference to the start (the "head") of a sequence while iterating through the rest of it, the JVM cannot garbage collect the elements already processed. This leads to a gradual increase in heap usage that eventually triggers an OutOfMemoryError (OOM), even if you believe you are processing data in a streaming fashion.

Diagnostic Mapping: Identifying the Cause

Use the following table to correlate observed memory behavior with the likely cause of retention.

Symptom	Likely Cause	Memory Behavior
Growth tied to request volume	Request-scoped binding retains lazy head	Incremental growth correlated with load
Growth only on large inputs	Chunked realization retaining 32-element blocks	Scales linearly with input size
Growth persists after completion	Lazy head stored in a cache, log, or global state	Persistent until the root reference is cleared

Ordered Diagnostic Checks

1. Heap Histogram Analysis: Run a heap histogram using a tool like jcmd <pid> GC.class_histogram. Look for a high count of clojure.lang.LazySeq, clojure.lang.Cons, or clojure.lang.ArrayChunk. If these dominate the heap, you have a sequence retention issue.
2. Stack and Binding Inspection: Examine thread stacks and local variable tables. Look for variables that bind the result of a map, filter, or range call and are passed into subsequent functions.
3. Code Path Audit: Search for patterns where a lazy sequence is assigned to a def at the namespace level or stored in a long-lived atom/ref. Any reference to the head of a sequence prevents the entire realized prefix from being collected.

Fixes Based on Findings

Depending on your diagnostic results, apply the corresponding fix to break the retention chain.

Finding: Local Binding Retention

If a local variable is holding the head while a loop processes the tail, replace the lazy pipeline with eduction or transducers. Eduction creates a reducible that does not cache elements during traversal.

Example: Retaining vs. Streaming

;; RISKY: 'res' retains the head, preventing GC of processed elements
(let [res (map inc (range 1000000))] 
  (doseq [x res] 
    (do-something x)))

;; SAFE: Transducer/into processes elements without caching the prefix
(into [] (map inc) (range 1000000))

Finding: Unintended Global Retention

If the sequence is defined at the top level, it will never be collected. Move the sequence generation inside the function that consumes it, or use a function that returns a new lazy sequence on every call.

Finding: Need for Eager Realization

If the full result is required and fits in memory, use doall or vec. This converts the lazy sequence into an eager collection, removing the overhead of LazySeq objects, though it allocates the entire result immediately.

Verification and Limitations

Practical Verification: To verify a fix, run a minimal reproducer in the REPL. Create a lazy sequence, bind the head to a variable, and run System.gc(). Use jcmd GC.class_histogram to confirm that LazySeq objects remain reachable. Apply the transducer fix and repeat the check; the LazySeq count should drop significantly.

Limitations: doall solves head retention but can cause an immediate OOM if the sequence is unbounded. Eduction is highly efficient for single-pass consumption but, unlike a vector, cannot be traversed multiple times.

Escalation Criteria

Escalate to JVM tuning or GC vendor support only if:

• A minimal reproducer confirms no LazySeq heads are being retained.
• The dominant objects in the heap dump are not Clojure sequence types.
• The memory leak persists even when using eager collections (vectors) and transducers.