Hinted Handoff vs Batch Log: Choosing the Right Strategy for Transient Node Failures in Multi‑Data‑Center Cassandra

Answer

Use hinted handoff as the primary recovery mechanism for transient node failures in multi-data-center clusters. Rely on the batch log only for logged batches that require atomicity across replicas. This is the recommended approach for production deployments spanning multiple data centers.

Key Configuration

Set max_hint_window_in_ms to cover your expected outage duration. Since Cassandra 3.0, the default is 3 hours (10,800,000 ms), not 60 seconds as in very old versions. For most multi-DC environments, 6–12 hours is appropriate. The hints_ttl setting was deprecated and removed in favor of max_hint_window_in_ms controlling hint retention.

Multi-DC Behavior

In NetworkTopologyStrategy deployments, hinted handoff operates primarily within the local data center. Hints for replicas in remote data centers are typically not stored unless explicitly configured via hinted_handoff_enabled_dc. Cross-DC hint delivery depends on replication settings and network topology, and enabling it increases coordinator load and inter-DC traffic during outages.

When to Disable Hinted Handoff

Disabling hinted handoff during prolonged outages prevents hint accumulation but shifts recovery burden to anti-entropy repair. When the node returns, data divergence persists until repair completes. Read repair may serve stale data in the interim, but eventual consistency is not violated long-term if repair runs to completion.

Configuration Steps

1. Verify current settings: Run nodetool getmaxhintwindow to confirm the actual hint window value in your cluster.
2. Monitor hint storage: Check hint directory size (typically $CASSANDRA_DATA/hints) and watch for pending hints metrics via nodetool tablestats system.hints.
3. Adjust if needed: Modify max_hint_window_in_ms in cassandra.yaml to match your expected outage duration, then perform a rolling restart.
4. Plan for batch log impact: If using heavy batch writes, monitor batchlog size via nodetool getbatchlogreplaythrottlekb to prevent write timeouts.
5. Schedule repair: Run scheduled repair weekly or after any outage exceeding the hint window.

Verification Commands

nodetool getmaxhintwindow
nodetool tablestats system.hints
nodetool getbatchlogreplaythrottlekb
nodetool repair --dry-run -pr

Important Considerations

• Batch log does not protect against data loss if the coordinator fails before the batchlog write completes. It only ensures atomicity across replicas once the batchlog entry is durable.
• Cross-DC hinted handoff should be evaluated carefully—enabling it increases inter-DC traffic during outages.
• The 60-second figure mentioned in some documentation reflects pre-3.0 defaults; verify your actual configuration.

One Missing Detail

What is your current max_hint_window_in_ms setting and typical network partition duration in your environment? This determines whether your existing configuration can handle your outage patterns or needs adjustment.