Understanding Cassandra Lightweight Transactions: When Linearizability Costs Too Much

The problem: conditional updates on an eventually consistent store

Cassandra’s default write path is fast because it only requires a quorum of replicas to acknowledge. That model gives you high throughput and low latency, but it also means two clients can overwrite each other’s changes without ever seeing a conflict. When you need a compare-and-set guarantee — for example, decrementing an inventory counter only if the current value is still positive — you need linearizable semantics on a single partition. That’s exactly what Lightweight Transactions (LWT) provide.

How LWT works under the hood

LWT runs a four-phase Paxos variant (prepare, propose, commit, learn) coordinated by the node that receives the CQL statement. A quorum of replicas must acknowledge each phase, so the critical path is roughly four times a normal quorum write. The ballot number for each partition lives in the internal system.paxos table; every new proposal increments that number, which serialises concurrent attempts on the same partition key.

Only the IF clause in CQL is supported for conditions. You can test equality, existence, or NULL on regular columns and primary-key components. Multi-partition atomicity is not provided — each partition runs its own Paxos round, so a batch that touches several partitions is not linearizable as a unit.

Performance reality check

• Latency: Expect ~4× the latency of an unconditional quorum write.
• Throughput: On a three-node cluster with SSDs, normal writes hit 50–100k ops/sec per node, while LWT tops out around 1–2k ops/sec per node. The gap widens with cross-datacenter latency or larger clusters.
• Contention: Hot partitions (counters, popular SKUs) cause cascade retries because every conflicting proposal must wait for the current ballot to finish. Tail latency can spike dramatically.

Because non-LWT writes bypass Paxos, mixing them with LWT on the same partition breaks linearizability. Also, TTL and USING TIMESTAMP clauses are ignored inside the IF condition; they only apply after the Paxos commit.

A worked example: inventory decrement

Suppose you have a table inventory with primary key sku and a column qty. You want to sell one unit only if stock is still available.

UPDATE inventory
SET qty = qty - 1
WHERE sku = 'ABC123'
IF qty > 0;

The driver sends this as an LWT. The coordinator runs the four-phase Paxos round on the partition that owns sku='ABC123'. If the condition holds, the decrement is applied and the statement returns [applied] = true. If another client already reduced the quantity to zero, the condition fails and the response contains the current row so the caller can decide what to do next.

Idempotent retries are safe: resending the exact same statement (same timestamp) will not double-apply because Paxos guarantees a single acceptance per ballot.

Operational tips and monitoring

• Enable tracing (TRACING ON) for a few LWT statements to see the four phases and measure the latency multiplier.
• Query system.paxos after a failed LWT to inspect ballot numbers and proposer host IDs — useful for debugging stuck proposals.
• Cassandra 4.0+ exposes a virtual paxos table with in-flight ballot state; monitor ballot-age metrics to catch long-running rounds early.
• Configure driver retry policies carefully: treat LWT timeouts as retryable only when the operation is idempotent, and disable speculative execution for LWT to avoid duplicate ballots.
• Remember that nodetool repair does not repair system.paxos; orphaned ballots persist until gc_grace_seconds expires.

When to reach for LWT (and when to walk away)

Use LWT when:

• You need a single-partition compare-and-set (e.g., claim a unique seat, decrement a counter, insert-if-not-exists).
• The contention on that partition is low to moderate.
• You can tolerate the 4× latency and reduced throughput.

Avoid LWT when:

• You need multi-partition atomicity — consider an external saga or a different data store.
• The partition is a known hotspot; instead shard the key or move the coordination to a dedicated locking service.
• Your workload is write-heavy and latency-sensitive; normal quorum writes will serve you far better.

Closing checklist

1. Identify the exact partition key that requires linearizability.
2. Prototype the LWT statement and run a quick trace to confirm the 4× latency on your hardware.
3. Load-test with cassandra-stress using a custom YAML that issues LWT updates; watch throughput collapse as concurrency rises.
4. Add alerts on system.paxos ballot age and driver-side LWT retry counts.
5. Document the decision: why LWT was chosen, the expected contention profile, and the fallback plan if latency becomes unacceptable.

Lightweight Transactions give you a powerful, Paxos-backed compare-and-set on a single partition, but they are a scalability lever, not a free lunch. Treat them as a targeted tool, monitor the ballot lifecycle, and you’ll keep both correctness and performance in balance.