Diagnosing Cassandra Read Latency Spikes from Tombstone Accumulation

Recognizable Condition

A sustained increase in read latency (consistently >200 ms) accompanied by a high TombstoneScanned metric in JMX indicates that read paths are spending excessive time skipping tombstones. This condition often appears as rising average latency in client‑side metrics while throughput remains stable or drops.

Cause / Diagnostic Table

Symptom	Likely Cause	Diagnostic Check
Read latency >200 ms, rising over minutes	Excessive tombstones from writes without TTL or frequent deletes	Query system.metrics.TombstoneScannedRate via JMX
Compaction lag visible in nodetool compactionstats	Compaction cannot keep pace with tombstone creation rate	Review pending compaction tasks and throughput
High write volume on a table	Write rate exceeds tombstone‑eligible expiration	Inspect recent writes per table (e.g., nodetool tablehistograms or system.size_estimates)

Ordered Checks

1. Check tombstone scan rate: jconsole or jcmd to read org.apache.cassandra.metrics:type=Table,scope=.name=,name=TombstoneScannedRate. Look for values consistently > 1 k/s as a warning sign.

2. Inspect compaction lag: run nodetool compactionstats. Note the Pending tasks and Total compaction throughput. If pending tasks grow while throughput stays low, compaction is lagging.

3. Review recent write volume: nodetool tablehistograms provides a write latency histogram; alternatively, query SELECT * FROM system.size_estimates WHERE keyspace_name='' AND table_name=''; to see total_data_size trends.

Fixes Tied to Findings

• Increase compaction throughput – for LeveledCompactionStrategy (LCS) raise the bucket size:

  ALTER TABLE . WITH compaction = {'class':'LeveledCompactionStrategy','sstable_size_in_mb':160};

  This allows larger SSTables, reducing compaction frequency and giving tombstones more time to be purged.

• Set appropriate TTL on writes to limit tombstone lifetime:

  INSERT INTO . (id, value) VALUES (?, ?) USING TTL 86400;

  Adjust the TTL to match your data‑retention policy.

• After any schema change (e.g., altering compaction strategy), run nodetool drain on the node before restarting to ensure memtables are flushed and reduce the chance of stale tombstones lingering in memory.

Escalation Criteria

Escalate to senior engineering or consider more aggressive remediation when either of the following persists:

• Average read latency > 500 ms for more than five consecutive minutes.
• TombstoneScannedRate remains > 10 000 per second after compaction‑throughput tuning and TTL adjustments.

Verification

1. Re‑run the same read workload (e.g., using cassandra-stress or your application’s test harness) and compare the average read latency before and after the changes.

2. Confirm that TombstoneScannedRate has dropped via JMX; a reduction of at least 50 % indicates the tombstone pressure is easing.

3. Check disk usage trends: nodetool cfstats . before compaction and after a major compaction cycle. Look for a decrease in SSTable count or stable On disk size while tombstone count declines.

Limitations and Practical Checks

• Changing compaction strategy triggers a rewrite of existing SSTables; this can cause temporary write stalls and increased I/O. Test the change in a staging cluster that mirrors production load first.
• Tombstone deletion does not immediately free disk space; the space is reclaimed only after the involved SSTables are compacted away. Monitor nodetool cfstats over several compaction cycles to confirm actual space reduction.
• If write volume is bursty, a static TTL may not match real‑time expiration needs; consider using time‑windowed TTLs or application‑level delete policies.

Rollback (if compaction strategy was altered)

1. Revert the table to its previous compaction definition:

   ALTER TABLE . WITH compaction = {'class':'', };

2. Run nodetool upgradesstables to rewrite SSTables using the original strategy.
3. Monitor nodetool compactionstats to ensure pending tasks return to baseline levels.