Geo‑replication design note for Apache Pulsar clusters

Problem

You need to mirror a namespace from one Pulsar cluster to another so that consumers in a remote region can read the same messages without changing application code. The goal is to understand the minimal architecture, where trust boundaries lie, how to verify it works, and what would force a redesign.

Requirements

• Two or more Pulsar clusters running the same major version (e.g., 2.10.x). Version skew can break ledger format compatibility.
• Inter‑cluster broker links secured with TLS; certificates must be trusted by both sides.
• WAN bandwidth sufficient to sustain the peak publish rate of the source namespace; otherwise replication lag will grow.
• Matching BookKeeper ledger configuration (ledger size, ensemble size, ack quorum) on both clusters so that ledger entries can be copied verbatim.
• Token‑based authentication enabled; namespace policies must allow the tenant to be replicated.

Smallest suitable design

The simplest viable topology is a primary‑standby pair where a single namespace is configured for geo‑replication. Pulsar’s built‑in replication mirrors ledger entries from the source BookKeeper cluster to the target without any extra middleware.

Configuration steps

1. On the primary cluster, add the standby cluster as a replication peer for the namespace:

pulsar-admin namespaces set-geo-replication-clusters \
    --clusters , \
    /

2. Verify the peer list:

pulsar-admin namespaces get-replication-clusters /

3. Ensure TLS is configured in broker.conf on both clusters:

brokerServiceTlsEnabled=true
webServiceTlsEnabled=true
 TLS certificate paths set appropriately

Run the commands from any broker host or a machine with the pulsar-admin CLI installed. The user must have superuser privileges or be a tenant admin with the produce and consume permissions on the namespace.

Trust and data boundaries

Replication traffic travels over the TLS‑encrypted broker links, so data never leaves the Pulsar trust domain. Access is governed by:

• Token‑based auth: the token presented by the broker must be valid for the tenant on both clusters.
• Namespace policies: only tenants whose namespace replication is enabled can have their ledgers mirrored.

Application‑level data is not exposed; if end‑to‑end encryption beyond TLS is required, the application must encrypt before publishing.

Operational checks

After configuration, monitor the following:

• Replication lag: pulsar-admin namespaces get-replication-lag / returns the number of ledger entries lagging behind the primary.
• BookKeeper ledger sync: on each cluster run bookkeeper shell ledgerinfo to confirm the ledger size matches.
• Broker‑reporter metrics: watch replication_rate (messages/sec) and replication_backlog (pending entries). Alerts should trigger if backlog exceeds a threshold (e.g., 5 minutes of lag at peak rate).

Verification example

1. Produce a burst of messages on the primary using pulsar-perf produce (e.g., 100 k messages at 10 k msg/s).
2. Immediately run the lag command; you should see a non‑zero lag that decreases as the standby catches up.
3. Consume from a subscription on the standby with pulsar-perf consume and verify that the received count matches the produced count after the lag drops to zero.
4. Optionally, compare ledger sizes: bookkeeper shell ledgerlist on both clusters should show similar total storage for the replicated namespace.

These steps do not guarantee zero loss; they only confirm that the replication mechanism is functioning as designed.

Failure modes

• WAN partition: replication stalls, backlog grows until the link is restored. If the backlog exceeds BookKeeper disk capacity, the standby may start discarding old ledger entries, causing gaps.
• Broker crash on either side: replication pauses until the broker restarts and re‑establishes the TLS link. Messages already acknowledged on the primary are safe; in‑flight messages may be retransmitted.
• Schema mismatch: if the topic has a schema and compatibility mode is set to INCOMPATIBLE, replicated messages that fail schema validation are dropped. Enable ALLOW_TRANSITIVE or BACKWARD compatibility to avoid loss.
• Version skew: running different major versions can lead to different ledger formats; replication may silently fail because the standby cannot interpret entries.

When the design would change

• Need for active‑active read/write** (multi‑master) across regions: you would configure per‑topic bidirectional replication or use Pulsar IO connectors to write to both clusters.
• Regulatory mandate for end‑to‑end encryption** beyond TLS: application‑level encryption (e.g., using AES‑GCM) would be required before publishing.
• Ultra‑low latency SLA (sub‑second replication) that cannot be met over the existing WAN: you might provision dedicated low‑latency circuits, place edge clusters closer to producers, or adopt a store‑and‑forward pattern with a local buffer.
• Scaling to many namespaces with heterogeneous replication policies: a central replication manager or custom controller may replace the manual pulsar-admin approach.

Limitations and practical checks

The design assumes:

• TLS termination is handled by the brokers; if you use a terminating proxy, ensure it does not strip or alter the replication protocol frames.
• BookKeeper ledger configuration is identical; mismatched ledgerSizeLimit or ensembleSize can cause replication errors that appear as increased lag.
• Token validity periods are long enough to survive broker restarts; otherwise re‑authentication failures will appear as replication pauses.

To check that these assumptions hold in your environment:

1. Review broker.conf on both clusters for matching tlsCertificateFilePath, tlsKeyFilePath, and authenticationProviders.
2. Run bookkeeper shell ledgerinfo on a sample ledger from the primary and the standby; compare entrySize and lastAddConfirmed values.
3. Check token expiry with your identity provider; if tokens are short‑lived, configure the broker to refresh them automatically.

If any of these checks fail, the replication design must be adjusted before relying on it for production workloads.