Topic not found error during concurrent auto‑create attempts

Question

When the auto‑create topics feature is enabled, a producer or consumer that references a non‑existent topic triggers the broker to create the topic on first use. In a multi‑node cluster, several clients may simultaneously issue the same creation request. The broker does not document a locking or serialization strategy for these concurrent auto‑create attempts, and logs often contain warnings such as topic already exists when duplicates are detected.
The unresolved decision is whether the Pulsar broker guarantees atomic creation of a topic across the cluster, or if external coordination is required to prevent race conditions. This uncertainty can affect performance measurements, as duplicate creation attempts may introduce latency spikes or inconsistent metadata states.
1. Does Pulsar guarantee that only a single topic node is created when multiple brokers receive concurrent auto‑create requests?
2. Is there an internal lock or serialization mechanism in the broker that ensures atomic topic creation across a cluster?

Tasadduq Burney · Answer

Answer
Pulsar does guarantee that only a single topic node is created when multiple brokers receive concurrent auto‑create requests, and the guarantee is enforced by an internal serialization mechanism.
Confirmed facts

Topic metadata is stored in ZooKeeper; creating a topic uses a ZooKeeper create‑if‑not‑exist operation, which is atomic across the cluster.
Only the leader broker for the namespace bundle processes metadata changes; all auto‑create requests are forwarded to that leader, which serializes them via ZooKeeper.

Likely explanation
Because the ZooKeeper operation either succeeds (topic created) or returns NodeExists, brokers treat the latter as a harmless duplicate and log “topic already exists”. This ensures exactly one topic node ends up in ZooKeeper, so external coordination is not required.
Steps to verify in your cluster

Check your Pulsar broker version (pre‑2.0 may use a different store).
Enable debug logging or watch the broker log while starting several producers/consumers against a non‑existent topic; you should see “topic already exists” warnings but no errors.
Use pulsar-admin namespaces get-bundle-leader to identify the bundle leader broker.
Inspect ZooKeeper directly: zkCli.sh -server :2181 ls /topics/// and confirm only one znode for the topic.

If your broker version is older than 2.0, please let me know so I can advise on any additional coordination needed.

Topic not found error during concurrent auto‑create attempts

1 answer

Answer

Confirmed facts

Likely explanation

Steps to verify in your cluster

Your perspective belongs here.

0 question comments