Cutting Kafka Rebalance Disruption with CooperativeStickyAssignor

The rolling restart tax

You push a minor bugfix to your consumer service. One pod restarts. Suddenly every consumer in the group pauses, every partition is reassigned, and lag spikes across the entire topic. If you run a 24-partition topic with three consumers, a rolling deployment of all three instances triggers three separate stop-the-world events. That is the default behavior of eager assignors such as RangeAssignor and RoundRobinAssignor: on any membership change, every consumer revokes all of its partitions, commits offsets, waits for the group to settle, and then receives a brand-new assignment.

An assignor is simply the client-side algorithm that decides which consumer owns which partitions. The broker does not pick it; every client in the group does.

What cooperative rebalancing changes

Since Kafka 2.4, the CooperativeStickyAssignor offers an incremental alternative. Instead of revoking everything, consumers only revoke the specific partitions that must move to another member. In the same 24-partition, three-consumer example, restarting one consumer means that member revokes roughly eight partitions while the other two keep processing their current assignments. The group still rebalances, but the blast radius shrinks from “all partitions, all consumers” to “only the partitions in flight.”

Behavior	Eager assignors	CooperativeStickyAssignor
Partitions revoked on membership change	All partitions, all consumers	Only partitions that must move
Consumers paused during rebalance	Entire group	Only members losing or gaining partitions
Convergence	Single round	Possibly multiple rounds; brief imbalance expected
Mixed assignors in same group	N/A	Falls back to eager full revocation

Convergence is gradual. The group may need multiple rebalance rounds before ownership stabilizes, so brief imbalance and temporarily idle partitions are expected. This is not a bug; it is the assignor trading immediate symmetry for reduced disruption.

Configuration and verification

Because the assignor is a client decision, every instance in the same group.id must use the same strategy. A typical Java consumer configuration looks like this:

props.put("group.id", "order-processor");
props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.put("enable.auto.commit", "true");

To confirm the behavior during a rolling restart, describe the group before and after the deployment. Run the following from any host with the Kafka CLI tools and network access to your brokers. The executing user needs DESCRIBE permission on the consumer group.

kafka-consumer-groups.sh --describe --group <GROUP_ID> --bootstrap-server <BOOTSTRAP_SERVERS>

Expected check: after one instance restarts, partition ownership should change only for the restarted consumer. If ownership shifts across all three consumers, the group has fallen back to eager rebalancing.

For deeper diagnostics, enable DEBUG logging for org.apache.kafka.clients.consumer.ConsumerCoordinator in your consumer service. Look for incremental revoke and assign messages rather than a global revocation of every partition.

Trade-offs and sharp edges

Cooperative rebalancing is not free, and several conditions can silently disable it.

• Mixed assignors break cooperation. If any member in the same group.id uses an eager assignor such as RangeAssignor or RoundRobinAssignor, the group falls back to full revocation of all partitions. Standardize the strategy across every service instance.
• Version alignment matters. The feature requires Kafka 2.4+ clients and compatible broker protocol versions. Mixed client versions in one group can also trigger eager fallback.
• Fewer revokes is not zero downtime. Consumers still pause briefly while revoked partitions are closed and offsets are committed.
• Detection speed is unchanged. session.timeout.ms and heartbeat settings still control how quickly a dead or stalled member is removed from the group.
• Large partition counts still cost time. Slow assignment logic can extend rebalance duration even without full revocation, so monitor rebalance time and lag during deployments.

Finally, cooperative rebalancing reduces reassignment disruption; it does not provide exactly-once processing. That still requires idempotent producers, acks=all, sufficient min.insync.replicas, and committed-read isolation where needed.

Before you rely on it

Treat cooperative rebalancing as a deliberate engineering decision, not a default upgrade. Verify that all clients and brokers are on compatible versions, confirm that every instance uses CooperativeStickyAssignor, and run the kafka-consumer-groups.sh check during a staging rollout before you trust it in production. Watch per-partition lag throughout the deployment: on partitions not owned by the restarting instance, lag should stay bounded. If it does not, or if the group reverts to eager behavior, fix the configuration mismatch before proceeding.