Consumer group rebalance pauses during a zero-downtime Kafka application migration

Question

A small Kafka consumer application is being moved to a new deployment, and the goal is to keep message processing continuous while old and new instances briefly share the consumer group. Each membership change in that window triggers a rebalance, and the assignment protocol decides how much of the group stops consuming.

Under an eager assignor such as RangeAssignor, any join or leave revokes every partition from every member until reassignment completes. The cooperative-sticky assignor, available in Java clients since Kafka 2.4, revokes only the partitions that actually move, which narrows the gap but pushes more work onto the application: revoke and assign callbacks must tolerate incremental changes, and duplicate processing remains possible.

The unresolved part is offset commit timing around revocation. Committing before processing finishes can lose records if a partition is handed off mid-batch, while committing afterward invites duplicates during the rebalance window. A slow rebalance can also push a member past max.poll.interval.ms, prompting eviction and further rebalances, and mixed client versions or differing partition.assignment.strategy settings within one group can change rebalance behavior.

Which assignment strategy fits a short rolling migration of a small consumer group? How should offset commits be timed relative to partition revocation so the transition causes neither message loss nor unbounded duplicates?

Tasadduq Burney · Answer

Answer
For a short rolling migration of a small consumer group, the org.apache.kafka.clients.consumer.CooperativeStickyAssignor is the appropriate assignment strategy. Offsets should be committed synchronously inside the ConsumerRebalanceListener.onPartitionsRevoked callback before the partition is released.
Confirmed facts

CooperativeStickyAssignor performs incremental, cooperative rebalancing: only partitions that actually move are revoked, reducing the processing gap.
Eager assignors (e.g., RangeAssignor) revoke all partitions from every member on any join/leave, causing a stop‑the‑world effect.
Committing offsets after processing finishes can cause duplicates during the rebalance window; committing before processing finishes can lose records if a partition is handed off mid‑batch.
To avoid message loss, offsets must be persisted before the revoked partition is no longer owned by the consumer.
Keeping the batch size small and making processing idempotent limits the impact of any unavoidable duplicate processing.
All members in the group must use the same partition.assignment.strategy setting; otherwise the group falls back to the least common denominator strategy.

Steps for this case

Set the consumer property partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor on every instance (old and new).
Implement a ConsumerRebalanceListener:

In onPartitionsRevoked, synchronously commit offsets for the partitions being revoked (use commitSync or transactional commit if applicable).
Optionally log the revoked vs. retained partitions for verification.
In onPartitionsAssigned, resume processing; no special commit is needed here.

Ensure max.poll.interval.ms is larger than the expected worst‑case rebalance time (including the synchronous commit). If it is not, increase it to avoid premature eviction.
Keep the consumer poll batch size modest (e.g., max.poll.records) to limit the amount of data that could be duplicated if a crash occurs between processing and committing.
Make the message‑processing logic idempotent (e.g., deduplicate by primary key or use exactly‑once semantics) so any occasional duplicate does not cause corruption.
Run the migration: start new instances, let them join the group, observe that only moving partitions are revoked, then shut down old instances after their partitions have been transferred.

Verification

Check broker logs for join-group and sync-group requests; they should show that not all members lose all partitions simultaneously.
Enable debug logging on the consumer (org.apache.kafka.clients.consumer) to see which partitions are listed in onPartitionsRevoked and confirm they match the expected moving set.
Monitor offset commit latency; ensure commits complete well before max.poll.interval.ms expires.

Missing diagnostic detail
If you are unsure whether the current max.poll.interval.ms setting accommodates the rebalance plus commit time, please provide that value so we can confirm it is sufficient or advise an increase.

Consumer group rebalance pauses during a zero-downtime Kafka application migration

1 answer

Answer

Confirmed facts

Steps for this case

Verification

Missing diagnostic detail

Your perspective belongs here.

0 question comments