KRaft controller roles in Kafka 4.0: combined versus dedicated placement after ZooKeeper removal

Answer

The decision between colocating broker and controller roles or running dedicated controllers in Kafka 4.0 hinges on two interrelated families of metrics:

• Resource contention & latency – CPU, memory, and network utilisation on a broker that also hosts a controller can inflate request latency, especially for high‑throughput workloads. If a broker’s CPU % usage regularly exceeds 70–80 % or its inter‑broker network utilisation is above 80 % of link capacity, colocating a controller can become a bottleneck.
• Quorum resilience & blast radius – A single node that owns both data and controller responsibilities removes two failure points in one go. In a 3‑node controller quorum (the minimum for a majority), losing such a node removes two votes, causing the cluster to stall metadata writes until a new controller is elected or a new node is added.

When a workload is latency‑sensitive or has a strict metadata‑write SLA (e.g., <10 ms for topic creation), a dedicated controller with extra CPU headroom keeps election time and log replication within bounds. For smaller clusters or cost‑constrained environments, colocating can reduce operational overhead, but operators must weigh the risk of a single point of failure that can bring the cluster offline.

Verifying Custom ZooKeeper Znodes and ACLs Before Decommissioning

Kafka 4.0’s KRaft metadata log does not automatically import arbitrary ZooKeeper znodes. Operators must explicitly confirm that every ACL and any custom configuration znodes they rely on are present in the KRaft metadata store.

1. Export the ZooKeeper snapshot – e.g., zkCli.sh -server zk1:2181 ls /kafka-acls and dump /kafka-acls to capture ACL entries and any custom znodes.
2. List ACLs in KRaft –

   kafka-acls --list --authorizer-properties bootstrap.servers=broker1:9092
   

3. Export KRaft metadata –

   kafka-metadata-show --bootstrap-server broker1:9092
   

   This command prints all topics, configs, ACLs, and other resources that KRaft has persisted.
4. Compare snapshots – A simple diff of the ACL lists (or a script that checks each ACL entry) will reveal missing entries. Repeat the process for any custom znodes (e.g., /kafka-clusters, /kafka-configs) to ensure corresponding records exist in KRaft (often under configs or cluster‑metadata entries).
5. Only after the diff shows zero discrepancies should you proceed with ZooKeeper decommissioning.

Availability Trade‑Offs for Combined Roles

• Quorum loss – In a 3‑node controller quorum, a single node failure removes two votes, halting all metadata writes until a new controller is elected.
• Latency spikes – Controller election and log replication share the same disk and network resources as data replication, potentially delaying both metadata writes and data replication.
• Operational simplicity – Fewer nodes mean less configuration overhead and lower networking cost, but the risk of a single point of failure is higher.
• Recovery time – If a combined node recovers, it must catch up on the KRaft log, which can be slower if the node was offline for an extended period.

Dedicated controllers mitigate these issues by isolating the quorum logic from data traffic, ensuring that a broker failure does not affect controller availability and that controller elections remain fast.

Missing Diagnostic Detail Needed

To tailor the recommendation precisely, please provide the current number of controller voters configured in your cluster. This number determines how many combined nodes you can afford to lose before quorum is broken.