Cassandra Lightweight Transactions: Paxos-Based Compare-and-Set in Practice

What you get from a Lightweight Transaction

A Lightweight Transaction (LWT) lets you execute a compare-and-set on a single partition with linearizable semantics. The statement either succeeds atomically — returning applied = true — or fails and returns the current row so you can decide what to do next. No external lock service is required; Cassandra runs a Paxos round internally.

How the Paxos round works

When you append an IF clause to an INSERT, UPDATE, or DELETE, the coordinator becomes a Paxos proposer for that partition:

1. Prepare – coordinator sends a prepare request to a quorum of replicas.
2. Propose – if a quorum promises, the coordinator proposes the new value.
3. Commit – on acceptance, the coordinator sends a commit to all replicas.
4. Ack – replicas acknowledge; the coordinator returns the result.

Each phase requires a network round‑trip, so an LWT typically costs 2‑4× the latency of a normal write.

Worked example: conditional balance decrement

UPDATE accounts
SET balance = balance - 100
WHERE id = 'user123'
IF balance >= 100;

Run this in cqlsh (or any driver) against a Cassandra 3.11+ / 4.x cluster. On success you receive:

 [applied] | id      | balance
-----------+---------+---------
      True | user123 |     900

On contention (another client changed the balance first) you get:

 [applied] | id      | balance
-----------+---------+---------
     False | user123 |     950

The returned row reflects the state at the moment the Paxos round read the partition; it is not guaranteed to be the latest committed value.

Performance characteristics you should expect

• Latency: 4 round‑trips across a quorum. In a 3‑node cluster with 1 ms intra‑DC latency, a single LWT often takes 4‑8 ms vs. 1‑2 ms for a normal write.
• Throughput under contention: Failed attempts still consume coordinator CPU and network. Hot partitions (e.g., a single inventory counter) can become bottlenecks; throughput drops sharply once contention exceeds ~10 % of requests.
• Consistency level: LWT always runs at SERIAL consistency (quorum for prepare/propose, all for commit). You cannot lower it.
• Cassandra 4.0+ tuning: cas_contacts (default 2) and cas_retry_timeout (default 10 s) in cassandra.yaml let you limit the number of replicas contacted during prepare and the maximum retry window.

Common mistakes and how to avoid them

1. Read‑then‑LWT pattern

Reading the balance first and then issuing the LWT creates a race window. The read may be stale, and the subsequent LWT will fail anyway. Always embed the condition inside the LWT itself as shown above.

2. Using LWT for high‑throughput idempotent writes

If your operation is naturally idempotent (e.g., setting a column to a fixed value), a normal write with client‑side retries is cheaper. Reserve LWT for true compare‑and‑set logic.

3. Collections or UDTs in the condition

LWT on a table that stores collections or user‑defined types rewrites the entire collection on each attempt, amplifying write amplification and latency. Model such data in separate partitions if you need conditional updates.

4. Ignoring read‑after‑write visibility

After a successful LWT, a subsequent SELECT ... WITH CONSISTENCY QUORUM may still return the old value because some replicas have not yet applied the commit. Use CONSISTENCY SERIAL for the read that must see the LWT result.

Monitoring and diagnostics

Metric	Source	What to watch
LWT Latency (p50/p99)	nodetool tablestats or JMX org.apache.cassandra.metrics:type=Table,keyspace=*,table=*,scope=LWTLatency	Sudden increase signals contention or GC pauses.
LWT Contention (failed attempts / total)	Same sources, metric LWTContention	>10 % suggests schema redesign (bucketing, async compensation).
Paxos phase tracing	TRACING ON in cqlsh, then inspect system_traces.events for PAXOS entries	Confirms 4‑phase execution; missing phases indicate coordinator failures.

Verification steps you can run today

1. Start a 3‑node cluster (Docker or ccm) with Cassandra 4.1.
2. Create the accounts table and insert a row.

   CREATE TABLE accounts (id text PRIMARY KEY, balance int);
   INSERT INTO accounts (id, balance) VALUES ('user123', 1000);

3. Run the conditional UPDATE from the example; confirm applied = true and balance 900.
4. Run the same UPDATE concurrently from two terminals; one will return applied = false with the current balance.
5. Execute SELECT * FROM accounts WHERE id='user123' WITH CONSISTENCY SERIAL; — it must show 900. Repeat with QUORUM to observe possible stale read.
6. Enable tracing (TRACING ON) and re‑run the LWT; query system_traces.events for four PAXOS entries (prepare, propose, commit, ack).
7. Use cassandra-stress with the lwt profile to generate load, then check nodetool proxyhistograms for latency percentiles.

Permissions: All commands require a user with MODIFY on the keyspace and SELECT on system_traces. nodetool needs JMX access (typically the cassandra OS user).

Limitations to keep in mind

• LWT is single‑partition only. No multi‑row or multi‑partition atomicity.
• Failed LWT returns a row that may already be stale; implement exponential back‑off with jitter before retrying.
• Repair operations (especially incremental repair) can spike LWT latency because Merkle tree calculations compete for CPU and disk I/O. Schedule repairs during low‑traffic windows.
• Cassandra 5.0 (early access) reworks the Paxos implementation; behavior may differ. Validate on your target version before production rollout.

Quick checklist before you ship

• Condition expressed inside the IF clause — no separate read.
• Partition key design avoids hot spots (consider bucketing or sharding).
• Client uses SERIAL consistency for any read that must observe the LWT result.
• Monitoring alerts on LWT Contention > 10% and LWT Latency p99 > 50 ms.
• Tested on the exact Cassandra version you will run in production (3.11, 4.0, 4.1, …).