Diagnosing and Fixing Tombstone‑Induced Read Latency in Cassandra

Recognizable Condition

In production you may see a sudden spike in client read latency and timeouts on one or more tables, while write throughput remains unchanged. Logs often contain warnings such as Too many tombstones in a single partition and you may notice increased young GC pauses on the nodes that serve the affected ranges.

Common Causes

Cause	Typical Scenario
Large range deletes or frequent single‑row deletes	Deleting many rows in a single partition without compaction.
Size‑Tiered Compaction Strategy (STCS) with many small SSTables	High tombstone churn but compaction never consolidates them.
Expired TTLs that outlive gc_grace_seconds	Tombstones linger longer than expected.
Wide partition scans without clustering order	SELECT * FROM table WHERE partition_key = ? ORDER BY none.

Ordered Diagnostic Checks

1. Check Tombstone and Read Latency Metrics
   Run on each node that hosts the affected token ranges:

   # nodetool tablestats <keyspace>.<table> | grep -E "tombstone|latency"
   

   Look for a high tombstone count and elevated read latency (ms) percentages.

2. Inspect System Logs for Tombstone Warnings
   Search system.log for “Too many tombstones” or “tombstone failure threshold” messages. Note the timestamp and affected keyspace/table.

   # grep -i "tombstone" /var/log/cassandra/system.log | tail -n 20
   

3. Measure Partition Size & Row Count
   Sample a few partitions to see if they are unusually large:

   # cqlsh -e "SELECT count(*) FROM <keyspace>.<table> WHERE partition_key = ? LIMIT 1;"
   

   Replace ? with a representative token value.

4. Verify Compaction Strategy and Delete Patterns
   Check the table definition:

   # cqlsh -e "DESC TABLE <keyspace>.<table>";
   

   Look for compaction {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'} and review application logs for bulk delete operations.

5. Check GC and JMX Metrics
   From a monitoring tool or JMX, confirm that GC pauses correlate with the spike. A high Young GC pause time often indicates tombstone compaction pressure.

Remediation Tied to Findings

• Excessive Tombstones & STCS
  If tombstone count > tombstone_warn_threshold:

  # nodetool compact <keyspace>.<table>
  

  Running a major compaction forces all SSTables to merge, eliminating tombstones that have survived gc_grace_seconds. Risk: I/O load spikes; test on a staging node first.
• Time‑Series Deletes
  For tables with frequent TTL expirations or range deletes, switch to TimeWindowCompactionStrategy (TWCS):

  # ALTER TABLE <keyspace>.<table> WITH compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_unit': 'DAYS', 'compaction_window_size': 1};
  

  TWCS groups SSTables by time window, making tombstone cleanup predictable.
• Wide Partition Scans
  Add clustering order or use a secondary index to avoid full partition scans. If you must scan wide partitions, limit the result set with LIMIT or WHERE clauses that break the scan into smaller ranges.
• Shorten gc_grace_seconds (with caution)
  If you confirm that deletes are safe to purge earlier, reduce gc_grace_seconds and run a major compaction. Do NOT lower this value unless you’re certain no node still holds pending writes for those rows.
• Tombstone Threshold Tuning (temporary)
  To observe the effect without changing delete patterns, raise thresholds:

  # nodetool cfstats <keyspace>.<table> | grep -i "tombstone"
  # nodetool setconfig tombstone_warn_threshold=200000
  # nodetool setconfig tombstone_failure_threshold=400000
  

  These changes are temporary and revert on restart; use them only for diagnostics.

Escalation Criteria

1. If read timeouts persist across multiple nodes after major compaction, the issue may be a data model problem or network partition. Escalate to the cluster operations team.
2. Persistent tombstone growth despite compaction indicates a deeper delete pattern. Consider redesigning the application to batch deletes or use TTLs.
3. Data loss risk when lowering gc_grace_seconds or performing forced compaction on a large dataset. Escalate to the data governance team before proceeding.
4. Schema changes that require a rolling restart (e.g., changing compaction strategy) should be coordinated with application downtime windows. Escalate to the release management team.

Practical Verification Checklist

• Run nodetool tablestats before and after remediation and confirm tombstone count drops.
• Monitor client read latency percentiles (p95, p99) via your APM or Prometheus for at least 24 h post‑fix.
• Verify that system.log no longer contains tombstone warnings for the affected table.
• Check that GC pause metrics return to baseline and that compaction backlog clears.

Limitations & Cautions

• Behavior varies between Cassandra 3.x and 4.x; always verify the version‑specific defaults for tombstone_warn_threshold and tombstone_failure_threshold.
• Major compaction can cause write stalls if the node is under heavy load; schedule during low‑traffic windows.
• Do not hide symptoms by raising thresholds permanently; address the root delete pattern instead.