Cassandra Compaction for Time-Series: Why TWCS Wins Until Writes Aren't Ordered

The symptom: compaction that never catches up

A metrics table with a 30-day TTL often looks healthy for weeks and then doesn't. Disk usage climbs past the raw data size, nodetool compactionstats shows a backlog that never drains, and read latency drifts because each query has to check more files than it should. The instinct is to tune thresholds. The more useful move is to ask whether the compaction strategy matches the lifecycle of the data.

The argument here: compaction strategy is a data-lifecycle decision, not a knob. For append-only time-series with TTLs, TimeWindowCompactionStrategy (TWCS) is usually the right answer, and it breaks in one specific, predictable way when writes aren't monotonic in time.

An SSTable is an immutable on-disk file of sorted rows. Every write produces new SSTables; compaction merges them so reads don't have to scan an ever-growing pile.

What each strategy is actually optimizing

The trade-off is structural. LCS keeps reads cheap by paying to rewrite data as it moves down levels, which is a bad deal when you never read the same row twice. TWCS keeps writes cheap by refusing to touch data it has already organized, which is a bad deal when data keeps changing.

How TWCS picks a window

TWCS assigns each SSTable to a window based on the maximum timestamp of the data it contains, not on the partition key. Within a window it merges SSTables using size-tiered-like rules. Across windows it does nothing: once a window closes, its SSTables are left alone unless a write lands in them.

That single behavior explains both the win and the failure. The win: compaction work stays proportional to the rate of new writes rather than to total data volume. The failure: a write whose timestamp falls into an old window reopens that window and forces compaction of cold data.

Window size is a retention and operations decision, not a performance tweak. compaction_window_unit and compaction_window_size together define the window. Pick them from how long you keep data and how you query it. Hourly windows keep SSTables small and make repair and bootstrap cheaper. Daily windows mean fewer, larger files and less compaction overhead. Very wide windows can produce multi-gigabyte SSTables that make streaming and repair noticeably more expensive.

A worked example: hourly windows with a 30-day TTL

Run this in cqlsh against a keyspace you own. Creating or altering a table requires CREATE or ALTER permission on the table or its keyspace.

CREATE TABLE metrics.raw_readings (\n  sensor_id     uuid,\n  bucket_day    date,\n  reading_time  timestamp,\n  value         double,\n  PRIMARY KEY ((sensor_id, bucket_day), reading_time)\n) WITH CLUSTERING ORDER BY (reading_time DESC)\n  AND default_time_to_live = 2592000\n  AND compaction = {\n    'class': 'TimeWindowCompactionStrategy',\n    'compaction_window_unit': 'HOURS',\n    'compaction_window_size': '1'\n  };

Two mechanisms are doing separate jobs here. The (sensor_id, bucket_day) partition key keeps partitions bounded so one sensor's history doesn't become a single enormous partition. The TWCS window is independent of that: it uses write timestamps. Aligning the two, a day-bucketed partition key with an hourly compaction window, is fine and common, but don't assume the partition key defines the window.

default_time_to_live is in seconds, so 2592000 is 30 days. It applies to rows written without an explicit TTL, and it's what lets TWCS drop whole SSTables once every cell in them has expired.

Checks worth running once the table is carrying traffic:

• nodetool tablestats metrics.raw_readings — look at SSTable count and space used. With TWCS and a steady write rate, SSTable count should track the number of open windows, not uptime.
• nodetool tablehistograms metrics.raw_readings — inspect the SSTable size distribution. If the largest files are much bigger than expected, the window is too wide.
• nodetool compactionstats — pending tasks should drain. A backlog that never clears is a signal to examine write patterns before touching thresholds.

The failure mode: writes into closed windows

TWCS assumes data is append-only and roughly ordered in time. Break that assumption and the strategy works against you:

• Late-arriving events. A backfill or a device that buffers for a day writes into yesterday's window. That window reopens and gets compacted again, repeatedly if the backfill is spread out.
• Updates to old rows. Any update to a row in a closed window is a new write with a new timestamp in that window. If you regularly update historical rows, TWCS is the wrong choice and LCS handles that access pattern better.
• Tombstone accumulation. TTL expiry and deletes generate tombstones, and a window full of them can stall compaction. nodetool tablestats reports estimated tombstone drop times; watch that value instead of guessing. unchecked_tombstone_compaction can force progress, but it can also resurrect data if repair has been inconsistent, so treat it as a last resort rather than a default.

Changing strategy on a live table

ALTER TABLE ... WITH compaction = {...} marks existing SSTables for rewrite. On a large table that is a substantial I/O event, so schedule it and watch nodetool compactionstats rather than assuming it is free. For a zero-downtime change, the safer pattern is a new table with the target strategy plus dual writes, then a backfill and a cutover.

What to do next

1. Write down your retention period and your query pattern. Those two facts determine the window; a benchmark does not.
2. Set the window from retention: hourly if you keep days, daily if you keep months. Smaller windows cost more files, larger windows cost more per-file streaming and repair.
3. Test with cassandra-stress against a staging cluster using a write profile that matches your real arrival pattern, including late arrivals. Capture nodetool compactionstats before and after.
4. If you cannot guarantee monotonic writes, choose LCS or STCS and accept the trade-off deliberately rather than discovering it in production.

Version note: TWCS options and compaction visibility have changed across 3.x, 4.0/4.1 and 5.0. Confirm exact option names and any compaction-related virtual tables against the documentation and release notes for the version you run before relying on them. Several details in this area are version-specific and are worth a check before you commit a schema change.