Embedded ClickHouse Keeper vs External ZooKeeper for Operational Isolation on Node Failure

Answer first

Confirmed: Both embedded ClickHouse Keeper and external ZooKeeper require a majority quorum for DDL and replica metadata changes. Reads and writes on existing table parts can continue without quorum after cluster initialization. Embedded Keeper runs in‑process with the ClickHouse server (introduced v21.8, GA v22.3) and shares CPU, memory and disk I/O with query workloads. External ZooKeeper runs in a separate JVM and communicates over the network, isolating coordination traffic.

Likely explanation: If a ZooKeeper‑to‑Keeper migration is interrupted while the cluster is changing its quorum configuration, some nodes may retain the old ZooKeeper quorum view while others have switched to the new Keeper quorum. This can cause temporary metadata inconsistency for DDL and replica metadata until the quorum views are reconciled. The documentation does not guarantee that a permanent split‑brain will occur, but manual quorum reconfiguration (e.g., via clickhouse keeper commands) may be required to restore consistency.

Version support for migration tooling

Confirmed: Migration from ZooKeeper to embedded Keeper is supported starting with ClickHouse v21.8, when Keeper was introduced, and the tooling is present in all releases that include GA Keeper (v22.3 and later).

Likely explanation / requires verification: The exact officially supported source ZooKeeper version to target embedded Keeper version pairs are release‑specific. You must consult the ClickHouse release notes or the clickhouse keeper migrate documentation for your current versions to confirm compatibility; unsupported major version jumps may lead to migration failure or lingering metadata divergence.

Co‑location and leader election under resource pressure

Confirmed: Embedded Keeper shares CPU, memory and disk I/O with the ClickHouse query workload, whereas external ZooKeeper runs in a separate JVM and network hop.

Likely explanation: Under CPU or I/O pressure, the in‑process Keeper can experience higher heartbeat latency variance compared to an isolated ZooKeeper ensemble. This variance can affect the timing of leader elections and, in extreme cases, increase the chance of a false leader election. ClickHouse’s internal heartbeats are designed to tolerate moderate load variation, so the risk is workload‑dependent and should be evaluated with staging benchmarks.

Steps needed for this case

1. Confirm the current ClickHouse version and ZooKeeper ensemble version on all nodes.
2. Verify that your version pair is listed as supported for clickhouse keeper migrate in the release notes.
3. Perform a pre‑migration quorum check: ensure all nodes report the same quorum state for the current coordination service.
4. Test the migration in a non‑production cluster, simulating node failure and resource pressure during the dual‑write and cut‑over phases.
5. After migration, verify that every node reports the same Keeper version and quorum configuration and that DDL progresses normally.

Missing diagnostic detail that changes recommendation: What ClickHouse version and ZooKeeper version is currently running in the cluster? The supported migration path and risk profile depend on that pair.