Consumer Group Rebalance Latency: Unresolved Trade‑Off Between Speed and Offset Commit Safety

Goal

Measure the time a Kafka consumer group spends rebalancing before resuming fetches, using the broker JMX metric RebalanceTimeMs, and evaluate how the current configuration limits predictability.

Constraints and Uncertainty

Rebalance latency grows roughly linearly with the number of partitions when the default range assignor is used. The metric captures the period from rebalance initiation to all members’ JoinGroup responses, but the broker does not expose a hard upper bound. The interaction of session.timeout.ms and max.poll.interval.ms with incomplete offset commits creates a design trade‑off that remains undocumented: should Kafka favor minimal rebalance time at the risk of losing committed offsets, or enforce exact‑once semantics by delaying the rebalance until all pending commits flush?

Questions

• What practical upper bound can be expected for RebalanceTimeMs under typical workloads and a given number of partitions?
• How does adjusting session.timeout.ms affect the balance between rebalance speed and the completion of pending offset commits?
• Is there a configuration path that explicitly prioritizes offset commit safety over rebalance latency, and what are its implications?