Topic not found error during concurrent auto‑create attempts
0 reputation · 12 Aug 2022, 14:56 UTC
When the auto‑create topics feature is enabled, a producer or consumer that references a non‑existent topic triggers the broker to create the topic on first use. In a multi‑node cluster, several clients may simultaneously issue the same creation request. The broker does not document a locking or serialization strategy for these concurrent auto‑create attempts, and logs often contain warnings such as topic already exists when duplicates are detected.
The unresolved decision is whether the Pulsar broker guarantees atomic creation of a topic across the cluster, or if external coordination is required to prevent race conditions. This uncertainty can affect performance measurements, as duplicate creation attempts may introduce latency spikes or inconsistent metadata states.
1. Does Pulsar guarantee that only a single topic node is created when multiple brokers receive concurrent auto‑create requests?
2. Is there an internal lock or serialization mechanism in the broker that ensures atomic topic creation across a cluster?