Consumer rebalance triggered by max.poll.interval.ms timeout during long-running record processing

When a Kafka consumer processes record batches that exceed the configured max.poll.interval.ms (default 5 minutes), the consumer proactively leaves the group and triggers a rebalance. This behavior is by design to detect stalled consumers, but it creates a tension with workloads that legitimately require extended per-record processing time, such as complex transformations, external API calls, or synchronous I/O.

The primary bounding knobs are max.poll.records (default 500) and max.poll.interval.ms. Reducing max.poll.records lowers the worst-case processing time per poll, yet it also increases poll frequency and may reduce throughput. Increasing max.poll.interval.ms accommodates longer processing but delays failure detection and can prolong the impact of a genuinely stuck consumer. Broker-side limits like fetch.max.bytes (default 50 MB per partition) and fetch.max.wait.ms (default 500 ms) further constrain the data volume per fetch request, but they do not bound processing time.

Exactly-once semantics (enable.idempotence=true with a transactional.id) prevent duplicate processing after a rebalance, yet they add latency and require idempotent downstream sinks. Without them, a timeout-induced rebalance can cause records to be reprocessed.

What heuristic or monitoring approach can determine a safe max.poll.records value for mixed-latency workloads without empirical trial and error? How should max.poll.interval.ms be sized relative to observed p99 processing latency to avoid spurious rebalances while still detecting true stalls? Does the introduction of KIP-626 (incremental cooperative rebalancing) change the risk profile of tuning these parameters in Kafka 3.6+?