Understanding Pulsar's Broker‑BookKeeper Separation: Design, Boundaries, and Failure Handling

Requirements

Apache Pulsar must support independent scaling of compute (message routing) and storage (durable log), provide strong consistency for replicated topics, and enforce authentication/authorization before data reaches the storage layer. The system should also tolerate node failures without losing messages or causing head‑of‑line blocking.

Smallest Suitable Design

The minimal architecture that satisfies these requirements consists of three layers:

1. Broker – stateless service that owns topics, handles producer/consumer connections, and forwards read/write requests to BookKeeper.
2. BookKeeper – distributed log service that stores message entries in segmented ledgers across a set of bookies.
3. ZooKeeper – coordination service that stores broker‑to‑topic ownership metadata, ledger fencing information, and cluster configuration.

In a standalone deployment you can run all three processes on the same host, but for production they are typically deployed on separate node groups to allow independent scaling.

Trust and Data Boundaries

The trust boundary is placed at the Broker interface:

• Clients authenticate (e.g., via TLS or JWT) and are authorized by the Broker before any request is forwarded to BookKeeper.
• The Broker validates topic names, checks namespace policies, and enforces quota limits.
• BookKeeper only sees authenticated Broker identities; it does not perform client‑level authentication.

Data never touches the Broker’s local disk; it is written directly to BookKeeper ledgers, which replicate entries across a write quorum (e.g., 3 bookies) and acknowledge based on an ack quorum (e.g., 2 bookies).

Operational Checks

To verify that the separation is functioning as intended, perform the following checks:

1. Process visibility – On each host, ensure the pulsar-broker, bookkeeper, and zookeeper JVMs are running.
2. Topic ownership – Run pulsar-admin topics stats persistent://my-tenant/my-namespace/my-topic and confirm the owner field shows a broker address.
3. Ledger layout – Inspect the BookKeeper storage directory (e.g., /data/bookkeeper/ledgers) and verify that ledger files are stored in sub‑directories named by ledger ID, confirming the segmented log layout.
4. Quorum verification – Describe a ledger with bin/bookkeeper shell describelog -ledgerid <ledger-id> and ensure the ensemble size matches the configured write quorum.
5. Latency hop – Use pulsar-admin topics stats to observe the averageMsgRateIn and averageMsgRateOut; a modest increase in latency compared to a local‑disk log is expected due to the network hop to bookies.

Failure Modes

Understanding how each layer fails helps you anticipate impact and plan mitigations:

Broker Failure

• If a broker crashes, ZooKeeper detects the session loss and triggers a ledger fencing operation.
• Topic ownership is transferred to another broker; producers and consumers reconnect transparently (assuming they are configured with multiple broker URLs).
• No data loss occurs because messages are persisted in BookKeeper.

Bookie Failure

• When a bookie becomes unavailable, writes that would have used it are redirected to remaining healthy bookies as long as the write quorum can still be satisfied.
• Reads follow the same rule; if ack quorum cannot be met, the broker returns a NotEnoughLedgerReplicasException to the client.
• Garbage collection must be configured (gcWaitTime, gcOverreplicatedLedgerWaitTime) to reclaim space from under‑replicated ledgers; otherwise disk exhaustion can occur.

ZooKeeper Failure

• A ZooKeeper outage blocks new broker‑to‑topic ownership changes and ledger fencing, but existing broker sessions continue until they expire.
• During the outage, producers can still publish to topics they already own, but new topic creation or broker scaling is paused.
• Running a replicated ZooKeeper ensemble (3 or 5 nodes) mitigates this risk.

Conditions That Would Change the Design

Certain workload or operational constraints may motivate a deviation from the pure broker‑BookKeeper split:

• Ultra‑low latency requirements – If sub‑millisecond write latency is critical, you might co‑locate a BookKeeper ledger directory on the same host as the broker and configure bookkeeperClientUseNetty=false to reduce stack hops, accepting higher operational coupling.
• Simplified operations – For small dev or test clusters, a single‑process standalone mode (bin/pulsar standalone) bundles all three layers, removing the need to manage separate services.
• Geo‑replication with asynchronous commit – When deploying a multi‑region cluster, you may relax the write quorum to a local region and rely on Pulsar’s geo‑replication engine to copy ledgers asynchronously, trading immediate consistency for lower cross‑region latency.