Consumer Group Rebalance Latency: Unresolved Trade‑Off Between Speed and Offset Commit Safety

Short answer

There is no broker-enforced upper bound on RebalanceTimeMs. The practical ceiling is the point at which a member is declared dead: roughly session.timeout.ms minus a safety margin. With the default range assignor, latency grows roughly linearly with partition count — expect on the order of a few hundred milliseconds for tens of partitions and several hundred milliseconds to around a second for a few hundred partitions, on healthy hardware. The trade-off you describe is real: Kafka does not have a switch that says "wait for pending commits before completing the rebalance," so you approximate that behavior with longer timeouts and manual synchronous commits.

What the metric measures — and what it misses

RebalanceTimeMs (on the consumer coordinator metrics, e.g. kafka.consumer:type=consumer-coordinator-metrics,client-id=<id>) measures the interval from rebalance initiation until all members have sent their JoinGroup responses. It does not include:

• time spent flushing pending offset commits to __consumer_offsets,
• time for the new assignment to be synced and partitions to resume fetching,
• any application-level drain of in-flight messages before rejoin.

So the metric systematically under-reports the true consumption pause. Treat it as a lower bound on user-visible stall time, not the stall itself.

How session.timeout.ms shifts the trade-off

session.timeout.ms is the broker-side dead-member detector. Lowering it forces faster failure detection and quicker rebalances — but it also shrinks the window in which a slow member can finish processing and complete an in-flight commitSync before being kicked out. The failure mode is not "lost committed offsets" (committed offsets in __consumer_offsets are durable); it is uncommitted progress: messages processed but not committed before eviction get re-fetched and reprocessed by whoever picks up the partition, producing duplicates.

Raising session.timeout.ms (and keeping heartbeat.interval.ms at roughly one-third of it) widens that commit window and raises observed RebalanceTimeMs proportionally. The caution: a high value delays detection of genuinely crashed consumers, so the group sits degraded longer.

The closest thing to a "safety-first" configuration

There is no single flag, but this combination prioritizes commit safety over rebalance speed:

1. enable.auto.commit=false and call commitSync() after processing each batch, so offsets are durable before the next poll.
2. isolation.level=read_committed on the consumer if you also use transactional producers, so duplicates from aborted work are not visible downstream.
3. A ConsumerRebalanceListener that calls commitSync() in onPartitionsRevoked — this is the key hook: it flushes pending commits during the rebalance, before partitions are reassigned.
4. Moderately raised session.timeout.ms and max.poll.interval.ms sized to your worst-case batch processing time.

Implication: you get at-least-once with rare duplicates (or effectively-once with transactions), at the cost of longer rebalances and slower crash detection.

Verifying it yourself

Since exact numbers depend on your version, hardware, and assignor, measure rather than trust any fixed figure:

# Enable JMX on the broker/consumer, then watch the coordinator metric
# while triggering a controlled rebalance (add/remove one consumer):
jconsole  # or your JMX scraper, MBean:
# kafka.consumer:type=consumer-coordinator-metrics,client-id=<id>

Run the experiment at 50 / 100 / 200 partitions, then repeat with session.timeout.ms at 10s / 30s / 60s, and finally with manual commitSync in onPartitionsRevoked. Compare RebalanceTimeMs and grep consumer logs for commit failures and duplicate deliveries. One caveat for skewed workloads: the range assignor can produce uneven assignments when subscriptions differ across members, which makes the linear latency model optimistic — consider cooperative-sticky if incremental rebalances matter more than raw rebalance speed.

Bottom line

Plan around session.timeout.ms as your de facto ceiling, expect linear-ish growth with partitions, and if commit safety wins over latency, use manual sync commits in the revocation listener plus longer timeouts — accepting slower failure detection as the price.