Clojure STM with Refs: Architecture Notes for Coordinated State

Requirements: When Optimistic Concurrency Fits

Clojure's Software Transactional Memory (STM) targets a specific niche: coordinated updates to multiple shared references within a single JVM process, where transactions are short, conflicts are rare, and contention is moderate. It does not address distributed coordination, blocking I/O, or cross-process synchronization. The model assumes you can express state changes as pure functions of the refs involved — side effects break retry semantics and are explicitly the caller's responsibility.

Smallest Suitable Design: dosync, Refs, and Validation

A transaction wraps a body of code in dosync. Every ref read or write inside is tracked. On commit, the runtime validates that no ref's version has changed since the transaction's snapshot. If validation fails, the transaction retries automatically (default limit: 10,000 attempts, then throws RetryEx). Refs hold any Clojure value; persistent collections (vectors, maps, sets) are idiomatic because structural sharing keeps snapshot costs low.

(def account-a (ref 1000))
(def account-b (ref 500))

(defn transfer [from to amount]
  (dosync
    (when (>= @from amount)
      (alter from - amount)
      (alter to + amount))))

Since Clojure 1.11, alter and ref-set accept IFn update functions, enabling transducer-like compositions:

(alter ref (comp (map inc) (filter even?) into []))

Trust and Data Boundaries

STM trusts that transaction bodies are pure functions of the refs they touch. Side effects — I/O, mutation of non-ref state, generating unique IDs, incrementing external counters — break retry semantics because the body re-executes on every retry. The io! macro marks blocks that must not run inside a transaction, but enforcement is by convention only; the compiler does not prevent it.

Data leaving a transaction (e.g., via alter returning a value) is a snapshot. Subsequent mutations by other transactions are invisible to the caller. This snapshot isolation is implemented by retaining history on each ref. By default, history is unbounded (ref-max-history is nil), which can cause memory pressure under long-running transactions. Set explicit limits per ref:

(def config (ref {} :max-history 5 :min-history 2))

Operational Checks: Instrumentation That Matters

Monitor clojure.lang.LockingTransaction via JMX or a Java agent. Key metrics:

• Retry count — high rates indicate contention hotspots
• Transaction duration — long transactions widen conflict windows
• Conflict rate — ratio of retries to commits

Enable the undocumented JVM flag -Dclojure.stm.stats=true for basic counters, or attach a profiler (e.g., clj-async-profiler) for histograms. For read-heavy workloads, tune ref-history-count and ref-max-history to bound memory while preserving snapshot isolation for concurrent readers.

Failure Modes and Mitigations

Failure Mode	Symptom	Mitigation
Live lock under high contention	Throughput collapses; threads starve	Reduce transaction scope; use commute for commutative ops (e.g., (commute ref conj item))
Unbounded retries	RetryEx after 10,000 attempts	Catch and escalate; redesign transaction or add backoff
Side effects in transactions	Duplicate I/O, emails, log entries	Discipline + io! assertions; move effects outside dosync
Memory pressure from history	Heap growth, GC pauses	Set :max-history per ref; avoid long-running transactions

Nested dosync blocks join the outer transaction — they do not create savepoints. An exception in any nested block aborts the entire transaction. This means you cannot partially commit; design transaction boundaries accordingly.

Conditions That Change the Design

Move away from STM when:

• Coordination spans processes — use core.async channels, actors, or a distributed consensus system
• Blocking I/O is required inside the critical section — STM cannot retry I/O safely; use core.async with thread pools or explicit locking
• Single-reference updates dominate — atoms (swap!, compare-and-set!) have lower overhead
• Profiling proves STM insufficient — only then consider LockingTransaction.runInTransaction with a custom Lock or java.util.concurrent.locks

Mixing STM with Java's synchronized or java.util.concurrent.locks risks deadlock; they do not compose. Pick one coordination model per critical section.

Verification Checklist

1. Stress test: N threads repeatedly dosync (alter ref1 inc) (alter ref2 dec) with varying N; measure throughput vs. single-threaded baseline. Expect near-linear scaling until contention dominates.
2. Side-effect isolation: Run a transaction that increments a volatile counter; confirm counter increments match commit count, not attempt count.
3. History tuning: Create a ref with :max-history 5, perform 100 updates in a long transaction, then check (ref-history-count ref) — should report ≤5.
4. Failure injection: Throw an exception mid-transaction; confirm no partial writes visible to other threads and no duplicate side effects if body is pure.

Version Assumptions

STM semantics have been stable since Clojure 1.0. The IFn update function support in alter/ref-set arrived in 1.11. Tooling (profilers, JMX exposers) may need updates for newer JDKs. The -Dclojure.stm.stats=true flag is undocumented and may change.