Consumer poll wakeup does not abort in-flight fetch requests during shutdown
0 reputation · 24 Apr 2025, 00:14 UTC
Goal: Identify a method to cancel in-flight fetch or produce operations in an Apache Kafka client without shutting down the consumer or producer, while ensuring that timeout settings such as max.poll.interval.ms, session.timeout.ms, request.timeout.ms and delivery.timeout.ms remain respected and that no unnecessary rebalances or data loss occur.
Constraints: The consumer.wakeup() mechanism only breaks the blocking poll call and does not interrupt network I/O that is already in progress; closing the client aborts any in-flight requests but may cause lost records or trigger a rebalance if done after timeout thresholds; the client lacks an external cancellation token or abort API for individual requests, making the behavior version‑dependent across major releases.
Questions:
- Is there a configuration or pattern that allows an application to abort an ongoing fetch or produce request immediately after invoking wakeup without closing the client?
- How do delivery.timeout.ms and request.timeout.ms behave when wakeup is triggered while a batch is being sent, and what is the effective latency observed?
- What are the trade‑offs between waiting for the configured timeouts to expire versus accepting request loss when using wakeup for a graceful shutdown?