Cassandra Read Latency Spikes from Tombstone Accumulation: A Diagnostic Guide
Read latency spikes on specific Cassandra tables with normal writes often signal tombstone accumulation from deletes and TTLs. This guide covers recognition, cause diagnosis, ordered checks with nodetool and logs, and fixes tied to findings with verification steps.
20 Jul 2025, 07:35 UTC
The recognizable condition
Read latency spikes and read timeouts on specific tables while writes stay normal. Clients report ReadTimeoutException or OperationTimedOut for a single keyspace.table. Server logs show tombstone warnings for that table, e.g. warnings about live tombstone scans or tombstone overflow. The problem is usually isolated to tables with frequent deletes or TTL expirations, not cluster-wide.
Tombstones are Cassandra's markers for deleted rows and expired TTL rows. They are not removed immediately; they persist for gc_grace_seconds to allow repairs to converge. Reads must scan past tombstones to confirm they are not newer than live data. When tombstone volume grows, read paths scan many dead rows before returning live rows.
Cause to symptom table
| Condition | Why it hurts reads | Typical signal |
|---|---|---|
| High delete or TTL churn with small page size reads | Each page request scans a fixed number of tombstones; small pages cause many round trips over the same tombstone set | Read timeouts on small fetch_size, normal writes |
| Small page size with large partitions | Large partitions contain many clustering rows; reads walk through tombstones across the partition | Latency grows with partition size, ReadStage high |
| Many SSTables, outdated compaction | Tombstones are spread across SSTables; read must merge tombstones from each | High Number of SSTables in nodetool tablestats, compaction pending |
| gc_grace_seconds too high or repairs missed | Tombstones linger longer than necessary after deletes | Tombstone count grows monotonically, no drop after major compaction |
Version note: tombstone threshold names and default values differ between Cassandra 3.x and 4.x. Log messages and read repair behavior also change with version. Verify defaults for your running version before adjusting thresholds.
Ordered checks
1. Confirm the table and collect logs
Run on a node with log access, OS user with read permission to Cassandra logs.
grep -i tombstone /var/log/cassandra/system.log | grep keyspace.tableLook for per-table warnings about live tombstone scans and tombstone overflow. Note timestamp correlation with latency spikes.
2. Inspect tombstone metrics
Run on any node as the cassandra OS user with nodetool access.
nodetool tablestats keyspace.tableCheck Number of tombstones, Live tombstones per read, and Average live tombstones per read. High values relative to live rows indicate scan cost. Also note Number of SSTables and Compaction pending tasks.
3. Check partition and read patterns
Review schema for TTL, delete patterns, partition key cardinality and clustering order.
cqlsh -e "DESCRIBE TABLE keyspace.table;"Requires CQL superuser for schema read. Look for default TTL, frequent DELETE statements, and wide partitions. Sample partition size with a bounded query using ALLOW FILTERING only for investigation, not production.
4. Compaction state
nodetool compactionstatsIdentify pending compactions for the keyspace.table. Many pending tasks or a large SSTable count suggests compaction backlog.
Fixes tied to findings
If logs show tombstone warnings and metrics show high tombstones per read
Temporarily raise thresholds for diagnosis only. This hides symptoms, it does not remove tombstones.
cqlsh -e "ALTER TABLE keyspace.table WITH tombstone_failure_threshold = 100000 AND tombstone_warn_threshold = 50000;"Risk: allows unbounded growth and masks root cause. Revert after investigation.
Increase client page size or use server-side paging to reduce per-request tombstone scans. Larger pages amortize scan cost over fewer round trips. Test in staging first.
If many SSTables and compaction pending
Schedule compaction after testing in non-production. Compaction strategy changes are not instant and increase IO.
nodetool compact keyspace tableRun on one node at a time. Major compaction rewrites SSTables and drops expired tombstones. Monitor disk IO and read latency during execution. Alternative: switch compaction strategy to one better suited for delete-heavy workloads, e.g., TimeWindowCompactionStrategy for TTL-heavy tables, after validation.
If TTL churn or delete pattern is the driver
Adjust data model to reduce churn. Options include shortening TTL granularity, batching deletes, or using a separate table for short-lived data. Changes require online schema migration planning.
Run repair to reconcile tombstones across replicas.
nodetool repair -pr keyspaceRequires node ownership and sufficient resources. Run during low traffic.
Verification
Compare nodetool tablestats tombstone metrics before and after changes to confirm reduction in tombstones per read.
Monitor client read latency percentiles and server logs for disappearance of tombstone overflow warnings.
Validate partition sizes with sampling queries and check SSTable count via nodetool tablestats to ensure compaction progress.
Limitations
Tombstone thresholds and log messages differ between Cassandra 3.x and 4.x. Behavior of read repair and speculative retry changes with version.
Compaction increases IO and can temporarily worsen latency. Do not run major compactions on production without capacity headroom.
Increasing thresholds hides symptoms rather than fixing root cause and can allow unbounded growth.
Escalation criteria
Escalate to storage engineering for coordinated repair and capacity review if:
- Persistent read failures across replicas after local fixes
- Tombstone counts grow despite regular compaction and repair
- Data loss suspicion after node loss or missed repairs
- Need for online schema change to alter compaction strategy or partition design
Provide log excerpts, nodetool tablestats output, and schema description for the affected table.
cassandra0 replies
A thoughtful contribution can make all the difference. Be the first to share one.