Cassandra Lightweight Transactions: Architecture Note on Requirements, Design Boundaries, and Operations

Problem: Need for Linearizable Compare‑and‑Set in a Single Partition

Applications that require a conditional update—such as "set this value only if it is currently X"—cannot rely on Cassandra’s eventual consistency guarantees alone. Without a mechanism that ensures the read‑modify‑write happens atomically, concurrent clients may overwrite each other’s changes, leading to lost updates or stale reads. Cassandra Lightweight Transactions (LWT) provide a linearizable compare‑and‑set primitive for a single partition, but they come with higher latency and operational overhead. This note outlines the requirements, the smallest viable design, trust and data boundaries, what to monitor, how failures manifest, and when the design should be reconsidered.

Requirements

• Atomicity: the condition check and the update must appear as a single indivisible operation to all clients.
• Linearizability: once an LWT succeeds, any subsequent read must see the updated value; a failed LWT must leave the value unchanged.
• Single‑partition scope: the key (partition key) must identify exactly one replica set; cross‑partition coordination is out of scope.
• Fault tolerance: the operation should tolerate the failure of a minority of replicas without losing correctness.

Minimal Design

The smallest CQL expression that triggers LWT is an INSERT or UPDATE statement with an IF clause. Example:

UPDATE users SET email = '[contact removed]' WHERE user_id = 123 IF email = '[contact removed]';

When executed, the coordinator (any node that receives the CQL request) runs a two‑phase Paxos protocol:

1. Prepare phase – the coordinator sends a Prepare message to a replica set (typically all replicas that own the partition). Each replica replies with a promise not to accept proposals with a lower ballot number.
2. Propose/Commit phase** – after gathering a quorum of promises, the coordinator sends a Propose containing the new value. Replicas accept the proposal if it matches their promise, persist it to the Paxos table, and reply. A quorum of accepts commits the value; the coordinator then informs the client of success ([applied] true) or failure ([applied] false).

If the condition fails (e.g., the current email does not match), the proposer still runs Paxos but proposes a no‑op; the client receives [applied] false.

Trust and Data Boundaries

LWT assumes:

• Replicas participating in the Paxos quorum are honest and follow the protocol; Byzantine behavior is not mitigated.
• The network can deliver messages reliably enough to achieve quorum within a reasonable time; partitions are treated as temporary unavailability.
• Data stored in the system.paxos table is trusted for recovery; it is not encrypted by default and should be protected via cluster‑level security (authentication, authorization, network encryption).

Consequently, LWT should be enabled only within a trusted administrative domain (e.g., a single VPC or a private network where node compromise is considered unlikely). Exposing LWT to untrusted clients does not increase safety beyond the normal Cassandra authentication model.

Operational Checks

To verify that LWT behaves as expected, monitor the following:

• Paxos table latency – query system.paxos for rows with state IN ('IN_PROGRESS', 'COMMITTED') and track the time between insertion and commit. Elevated times indicate quorum delays.
• Thread pool stats – run nodetool tpstats and observe the Paxos column. Pending tasks should stay near zero under normal load; a growing backlog signals that replicas are slow to respond to Prepare/Propose messages.
• Commit log and hinted handoff – LWT generates two writes per attempt (prepare log + commit log). Use nodetool compactionstats and watch for increased commit‑log size or a rise in hinted handoff during retries.
• Client‑side latency – compare the response time of LWT statements against regular writes (e.g., via cqlsh timing or application metrics). Expect 2‑4× higher latency; if the gap widens, investigate network jitter or GC pauses.

Example diagnostic query (run in cqlsh with a user that has SELECT on system tables):

SELECT partition_key, row_id, state, proposed_at, committed_at
FROM system.paxos
WHERE state IN ('IN_PROGRESS', 'COMMITTED')
ORDER BY proposed_at DESC
LIMIT 20;

Check that proposed_at and committed_at timestamps are close; a large gap suggests a stalled Paxos round.

Failure Modes

• Replica unavailable during Prepare – the coordinator can retry with another replica as long as a quorum can still be formed. If fewer than a majority of replicas respond, the operation fails with UnavailableException.
• Network partition – while the partition persists, any LWT requiring a quorum across the split will block or fail. Once connectivity is restored, pending proposals either commit or are cleared after a timeout (controlled by paxos_propose_wait_duration).
• Coordinator crash – if the coordinator fails after sending Prepare but before Propose, a new coordinator (the node that retries the client request) will re‑run the Prepare phase using the ballot numbers stored in system.paxos. No data loss occurs.
• Disk full on Paxos table – because each LWT attempt writes a row to system.paxos, sustained high‑rate LWT can fill the disk. Monitoring system.paxos size and setting appropriate TTL or purging old rows is essential.

When the Design Should Change

Reconsider using LWT if any of the following conditions arise:

• The workload requires conditional updates across multiple partitions; LWT cannot provide cross‑partition atomicity.
• Measured LWT latency exceeds the application’s SLA and retries due to transient unavailability are frequent.
• Operational metrics show sustained high pending Paxos tasks or commit‑log growth that impacts overall cluster performance.
• The cluster operates in a zero‑trust environment where replica honesty cannot be guaranteed (e.g., multi‑tenant public cloud with untrusted node operators).

In those cases, consider alternative patterns:

• Application‑level locking or a distributed consensus service (e.g., ZooKeeper, etcd) for cross‑partition transactions.
• Using BATCH with UNLOGGED for multiple mutations within a single partition when strict linearizability is not required.
• Designing the data model to avoid conditional updates (e.g., using time‑series or version vectors) and relying on eventual consistency with conflict‑resolution logic.

Practical Verification Steps

1. Create a test table: CREATE TABLE test_lwt (id INT PRIMARY KEY, val TEXT);
2. Run an LWT with consistency quorum: UPDATE test_lwt SET val = 'new' WHERE id = 1 IF val = 'old' USING CONSISTENCY QUORUM;
3. Check the result: the returned [applied] flag should be true if the precondition held, otherwise false.
4. Immediately query system.paxos for the partition key 1 to see a row transition from IN_PROGRESS to COMMITTED (or remain absent if the condition failed).
5. Run nodetool tpstats before and after a burst of LWTs; verify that the Paxos pending tasks return to baseline.
6. Simulate a replica failure: nodetool drain <node> (requires nodetool access, typically run as the Cassandra service user). Observe whether the LWT succeeds with the remaining replicas or returns UnavailableException when quorum cannot be met.

These steps confirm that the protocol progresses as designed and that observable metrics align with expectations.

Limitations

Even when functioning correctly, LWT has inherent trade‑offs:

• Throughput is limited by the two‑round‑trip Paxos exchange; expect far lower write rates than regular INSERT/UPDATE.
• Each attempt generates two commit‑log entries, increasing pressure on disk I/O and compaction.
• The feature does not solve the “write‑skew” anomaly across multiple rows; application logic must handle higher‑level invariants.

Monitoring the metrics described above provides a practical way to detect when those limits are being approached.

Conclusion

Cassandra Lightweight Transactions give a clear, linearizable compare‑and‑set primitive for single‑partition workloads, but they rely on a trusted replica set, incur noticeable latency, and generate extra I/O. By starting with the minimal CQL pattern, verifying Paxos table activity and thread‑pool health, and watching for the failure modes outlined, teams can operate LWT safely. When latency, cross‑partition needs, or trust assumptions shift, the design should be revisited in favor of alternative consistency patterns.