Kafka consumer group rebalancing behavior when max.poll.interval.ms times out during cancellation
0 reputation · 19 May 2024, 06:45 UTC
Problem Context
Apache Kafka consumers have built-in mechanisms for graceful shutdown through the consumer.wakeup() method, which throws WakeupException to interrupt the poll loop. However, the interaction between cancellation handling and rebalance timeouts requires careful consideration.
The max.poll.interval.ms configuration (defaulting to 7 minutes) defines the maximum time between consecutive poll() calls before the consumer triggers a rebalance. During application shutdown or cancellation scenarios, if the processing time exceeds this threshold, the consumer may leave the group unexpectedly.
Current documentation indicates that consumer.wakeup() provides a clean interruption mechanism, but the precise behavior when a wakeup occurs near or after the poll interval timeout is not clearly specified. The coordination between explicit wakeup calls, automatic timeout detection, and the resulting rebalance events needs clarification.
Key Questions
- How does Kafka client behavior differ when consumer.wakeup() is called within vs. outside the max.poll.interval.ms window?
- What is the exact sequence of events and state transitions when a timeout occurs during cancellation handling?
- Which configuration combinations ensure predictable shutdown behavior across different Kafka client versions?