Choosing the Right Compaction Strategy in Cassandra for Predictable Reads and Stable Writes

Problem & Takeaway

In a Cassandra table that receives high‑throughput writes and also serves frequent read requests, the compaction strategy determines how often data is reorganised on disk. Choosing the wrong strategy can cause unpredictable read latency, excessive disk usage, or even write failures. The key takeaway is that you should start with SizeTieredCompactionStrategy (STCS) during write‑heavy periods and switch to LeveledCompactionStrategy (LCS) once read latency becomes a Service Level Agreement (SLA) concern. This switch requires careful monitoring and sufficient free disk space.

Requirements

• Predictable read latency – 95th‑percentile reads must stay below 5 ms.
• High write throughput – >10 k writes/sec without timeouts.
• Bounded disk usage – growth rate <30% per month.
• Minimal operational overhead – no manual compaction triggers or repair storms.

Minimal Viable Design

Start with STCS. It groups SSTables by size, creating large files that are cheap to write. When read latency spikes, run an ALTER TABLE to switch to LCS, which keeps SSTables at a maximum of 64 MB and guarantees a bounded number of files per level. The migration is node‑local and does not affect cross‑node trust boundaries, but you must allow cassandra-stress or a similar tool to verify the transition.

# Run on a node with sufficient free space (at least 1.5× the table size)
ALTER TABLE keyspace_name.table_name \\
  WITH compaction = {'class': 'LeveledCompactionStrategy'};

Replace keyspace_name and table_name with your actual identifiers. Ensure the session has ALTER privileges. After the command, Cassandra will create new LCS SSTables while keeping the old STCS files until they are compacted into level‑0.

Trust & Data Boundaries

Compaction is performed on a per‑replica basis. It does not cross node boundaries, so data integrity between replicas is maintained by the normal repair process. However, the compaction process can temporarily double the disk footprint of a replica until the old SSTables are purged. This temporary spike is why you must monitor disk usage during the migration.

Operational Checks

1. Pending Compaction Tasks
   • Command: nodetool compactionstats
   • Expectation: PendingCompactionTasks drops to < 5 after migration.
   • Risk: If it stays high, disk space may be exhausted.
2. Compaction Executor Utilization
   • JMX MBean: org.apache.cassandra.metrics:type=CompactionExecutor
   • Metric: Utilization < 70% during steady state.
   • Risk: Sustained >70% can increase write latency.
3. Disk‑Space Growth
   • Command: nodetool info – check DiskSpaceUsed vs DiskSpaceAvailable.
   • Expectation: Growth rate <30% monthly.
   • Risk: If growth >50%, consider TTL or archiving.
4. Read Latency Histogram
   • Tool: cassandra-stress write‑read mix or real client metrics.
   • Metric: 95th‑percentile < 5 ms.
   • Risk: Latency >10 ms indicates compaction thrashing.

Failure Modes & Mitigations

• I/O Spikes During Compaction – can raise write latency. Mitigation: Use CompactionThrottleInMbPerSec in cassandra.yaml to cap I/O.
• Disk Exhaustion – compaction may abort if not enough space. Mitigation: Perform migration during low‑traffic windows and ensure at least 1.5× free space.
• Write Amplification with LCS – each write may touch multiple SSTables, increasing GC pressure. Mitigation: Tune gc_grace_seconds and monitor JVM pause times.
• Repair Storms – new LCS files can trigger large repair operations. Mitigation: Run nodetool repair with the –par flag to parallelise across nodes.

When to Re‑evaluate the Strategy

• Workload shifts to a time‑series pattern with heavy TTL usage → consider TimeWindowCompactionStrategy (TWCS) for efficient TTL expiration.
• Range scans dominate reads → LCS may still be preferable, but monitor CompactionExecutorUtilization closely.
• Regulatory requirements impose immutable storage → disable compaction entirely and rely on read‑only snapshots.
• Disk usage grows beyond 70% of capacity → revisit compaction strategy or add more nodes.

Concrete Example: Switching from STCS to LCS

Assume a 50 GB table on a node with 200 GB free space.

# Verify current strategy
DESCRIBE TABLE keyspace_name.table_name;

# Switch to LCS
ALTER TABLE keyspace_name.table_name \\\n  WITH compaction = { 'class': 'LeveledCompactionStrategy' };

# Validate change
DESCRIBE TABLE keyspace_name.table_name;

# Monitor compaction progress
nodetool compactionstats | grep -i 'PendingCompactionTasks'

After the command, watch for a temporary increase in DiskSpaceUsed up to ~75 GB. Once the migration completes, the table should have a steady 64 MB level‑0 file and predictable read latency.

Practical Verification Checklist

Check	Command / Tool	Expected Result
Strategy Field	DESCRIBE TABLE	LeveledCompactionStrategy
Pending Tasks	nodetool compactionstats	<5
Utilization	JMX CompactionExecutor.Utilization	<70%
Read Latency	cassandra-stress write‑read mix	95th‑pct <5 ms
Disk Growth	nodetool info	<30%/month

Conclusion

By starting with STCS for write‑heavy periods and transitioning to LCS when reads become critical, you balance write throughput, read latency, and disk usage. The transition is safe if you reserve enough disk space, monitor key metrics, and avoid frequent strategy changes. Adjust the strategy again only when workload patterns or compliance needs evolve.