SSTableLoader performance limits during large-scale snapshot restoration

Answer

The documented mechanisms to keep sstableloader from overloading a live cluster are twofold:

• Per‑stream throttling – the -t/--throttle flag caps the transfer rate in megabytes per second for each individual stream.
• Connection limiting – the -d/--connections flag (default 1) limits how many parallel streams a loader will open to a single target node.

In addition, Cassandra itself imposes a node‑wide bandwidth ceiling via cassandra.yaml:

• stream_throughput_outbound_megabits_per_sec (default 200 Mbps) throttles all outbound streaming traffic, including sstableloader, repair, bootstrap, and cross‑DC restores.
• The cross‑datacenter counterpart, inter_dc_stream_throughput_outbound_megabits_per_sec, applies to traffic that crosses data‑center boundaries.

Likely explanation vs. confirmed facts

Likely explanation: By capping both the per‑stream rate and the number of concurrent streams, the loader reduces instantaneous network and CPU load, allowing the cluster to continue serving reads and writes with acceptable latency during a large restore.

Confirmed facts: The sstableloader command-line interface exposes -t and -d; Cassandra’s default stream_throughput_outbound_megabits_per_sec is 200 Mbps, and the inter‑DC limit is configurable similarly.

Steps for this case

1. Determine the portion of your link you can spare for restore traffic. For example, on a 1 Gbps link you might reserve 30 % (≈300 Mbps) for the loader.
2. Convert that to megabytes per second: 300 Mbps ≈ 37.5 MB/s. Use the -t flag to enforce this per stream:

   sstableloader -t 37 -d 2 /path/to/snapshot 10.0.0.5

   This opens up to two concurrent streams per node, each limited to ~37 MB/s.
3. If you run multiple restore jobs or other streaming operations, adjust stream_throughput_outbound_megabits_per_sec in cassandra.yaml so the total outbound bandwidth stays below your link capacity. For a 1 Gbps link, you might set it to 800 Mbps to allow headroom for live traffic.
4. For cross‑DC restores, similarly tune inter_dc_stream_throughput_outbound_megabits_per_sec to match the inter‑DC link capacity.
5. Validate the configuration by running a small test restore and observing outbound traffic with nodetool netstats or a network monitor. Ensure the observed throughput does not exceed the configured limits.

Missing diagnostic detail

To fine‑tune -t, it would help to know the current peak outbound bandwidth of your nodes during a test run. If you can share that figure, I can suggest a precise throttle value that leaves enough headroom for live traffic.