Diagnosing Cassandra Write Latency Spikes Caused by Compaction Backpressure

Your Cassandra writes were running at 2–3 ms, and now they're sitting at 15–40 ms with no change in client traffic. If nodetool compactionstats shows pending compactions climbing while writes slow down, you're looking at compaction backpressure: the node can't merge SSTables fast enough, so each write has to contend with a growing pile of uncompacted data and saturated I/O. This guide walks through confirming that diagnosis, finding the specific cause, and applying the right fix — because the correct remedy depends on why compaction fell behind.

The recognizable condition

The typical signature is a combination of three observations on the same node:

• Write latency consistently above your threshold (for many clusters, sustained p99 above ~10 ms is worth investigating).
• nodetool compactionstats reports a growing number of pending compactions rather than a stable or shrinking count.
• Elevated CPU or disk utilization on the affected node, visible in iostat or your monitoring stack.

A single long-running compaction is normal. The warning sign is the trend: pending tasks accumulating over tens of minutes while write latency degrades in step.

Common causes at a glance

Ordered diagnostic checks

Run these on the affected node. nodetool commands need access to the node's JMX interface (typically run locally as the Cassandra user or an account with JMX credentials).

1. Confirm the backlog. Run nodetool compactionstats twice, a few minutes apart. Note pending tasks and the compaction throughput line. Rising pending counts confirm backpressure rather than a transient burst.

2. Check thread pool pressure. Run nodetool tpstats and look at MutationStage pending and blocked counts. Pending mutations alongside pending compactions means writes are queuing behind I/O, not just slow individually.

3. Measure the disk. Run iostat -x 5 (from the sysstat package) and watch await and %util for the data volume. Await times in the tens of milliseconds with %util pinned near 100 indicate the storage layer is the bottleneck, regardless of Cassandra settings.

4. Review the configuration. In cassandra.yaml, check compaction_throughput_mb_per_sec (16 MB/s is a common default in many versions) and concurrent_compactors. Also check the per-table strategy: DESCRIBE TABLE keyspace.table; in cqlsh shows the compaction options in effect.

5. Characterize the latency. Use nodetool tablehistograms keyspace table or client-side metrics to see the write latency distribution. A long tail with a healthy median suggests intermittent stalls; a shifted median suggests systemic saturation.

Fixes tied to findings

Compaction throughput is the limiter

If pending compactions are high but the disk has headroom (low await, %util well under 100), raise the throughput cap. You can test this without a restart:

nodetool setcompactionthroughput 32

This sets the rate to 32 MB/s on the local node until restart. If pending compactions start declining, make it permanent in cassandra.yaml and consider increasing concurrent_compactors (a common starting point is one per CPU core, capped by what your disks can serve). Risk: higher compaction throughput raises total I/O load and can degrade read latency if storage was closer to saturation than it appeared. Watch read latency in tpstats after the change.

STCS is drowning in small SSTables

Under sustained heavy writes, Size-Tiered Compaction can produce many small SSTables faster than it merges them. Options include raising min_threshold so compactions merge more SSTables per pass, or migrating the table to Leveled Compaction, which bounds read amplification more tightly for write-heavy workloads. Changing strategy is an ALTER TABLE in cqlsh, but be aware it triggers a rewrite of existing SSTables — expect a temporary spike in disk usage, CPU, and compaction work while the table is rewritten. Do this during low traffic and one table at a time.

LCS is doing too much leveling

If the table already uses LCS and compactionstats shows constant activity, a small sstable_size_in_mb (the 160 MB default in many versions) may be forcing frequent promotions. Increasing it reduces total compaction work per byte written, at the cost of larger individual compactions. Adjust via ALTER TABLE ... WITH compaction = {'class': 'LeveledCompactionStrategy', 'sstable_size_in_mb': '320'}; and observe.

The disk itself is saturated

No Cassandra setting fixes a disk that's already at its limit. Move to faster storage (or dedicated volumes for data and commitlog), rebalance data across nodes if the hot node holds disproportionate load, or throttle writes at the client. Check for non-Cassandra processes competing for the same volume.

Verifying the fix

• nodetool compactionstats shows pending compactions steadily declining and a compaction rate consistent with your new throughput setting.
• Write latency histograms (nodetool tablehistograms or client metrics) show the bulk of writes back under your threshold for at least 30 minutes.
• Read latency and overall throughput have not regressed — check ReadStage in tpstats and your client-side graphs.

When to escalate

Escalate beyond configuration tuning if write latency stays above threshold after two full compaction cycles following your changes, or if pending compactions keep rising despite increased throughput. At that point: add nodes to spread the write load; investigate hardware faults (disk SMART data, RAID controller errors, dmesg I/O errors); and review JVM/GC behavior, since long GC pauses can masquerade as compaction stalls. Version note: exact option names and defaults vary across Cassandra 3.x, 4.x, and 5.x — confirm against your version's cassandra.yaml documentation before changing values.