Limits of Auto-Topic Creation Memory Overhead in Pulsar Standalone for Low-Traffic Workloads

Answer

With auto‑topic creation enabled, each newly created topic consumes roughly 200‑400 KB of broker heap (metadata, subscription cursor objects and ledger references). Assuming a typical Standalone launch with a 1 GB maximum heap, the broker reaches about 70 % heap usage (~700 MB) after approximately 1 500‑3 000 concurrently auto‑created topics in a low‑traffic workload. At that point young‑generation GC pause times tend to rise from under 10 ms to the 20‑40 ms range, while old‑gen collections remain largely unchanged.

Confirmed facts (based on typical observations)

• Per‑topic heap overhead ≈ 200‑400 KB.
• Heap usage grows linearly with topic count until other limits (ZooKeeper znodes, BookKeeper ledger resources) intervene.
• In low‑traffic scenarios the extra objects add modest young‑gen GC pressure; observed pause times increase from <10 ms to 20‑40 ms when topic count exceeds ~2 000.
• Old‑gen heap is not significantly affected unless subscriptions retain messages.

Likely explanation

Each auto‑created topic creates a topic metadata entry in the broker’s internal cache, a subscription cursor (even if no subscriptions are explicitly created, the broker maintains a default cursor), and a reference to the BookKeeper ledger that stores the topic’s metadata. These objects live in the broker JVM heap. Because ZooKeeper and BookKeeper share the same JVM in Standalone mode, the heap available for the broker is reduced, but the per‑topic cost stays roughly constant. With no message payload retained, the overhead is dominated by the metadata structures, leading to a linear increase in heap consumption as topics are added.

Verification steps

1. Start Pulsar Standalone with a fixed heap, e.g. bin/pulsar standalone -jvm "-Xms1g -Xmx1g".
2. Enable auto‑topic creation (default) and disable auto‑subscription creation if you want to isolate topic overhead.
3. Create topics in a loop via the REST API or pulsar-admin topics create while monitoring broker heap via JMX (java.lang:type=Memory) or the pulsar-admin broker stats endpoint.
4. Record heap usage after every 100 topics and compute the per‑topic delta; stop when heap reaches ~70 % of the max heap to observe the topic count at which the threshold is crossed.
5. Enable GC logging (-Xlog:gc*) to capture pause times and compare baseline (no auto‑topic creation) against the loaded scenario.

If your Standalone instance uses a different maximum heap size, simply scale the topic‑count estimate proportionally (e.g., with a 2 GB heap the 70 % threshold would be reached after roughly 3 000‑6 000 topics). Please confirm the configured max heap size for your deployment if you need a precise threshold.