Question
Limits of Auto-Topic Creation Memory Overhead in Pulsar Standalone for Low-Traffic Workloads
Rahul Sage
0 reputation · 25 Jun 2023, 20:25 UTC
104.8K views
Loading…
0 reputation · 25 Jun 2023, 20:25 UTC
The goal is to quantify how enabling Auto‑Topic Creation influences broker heap consumption when running Pulsar in Standalone mode for a low‑traffic workload.
In Standalone mode the broker, ZooKeeper and BookKeeper share a single JVM, so any per‑topic data structures (metadata, cursors, subscription objects) directly affect the available memory. Low‑traffic workloads generate few messages, but the number of topics that may be created automatically is unpredictable, making it unclear at what point the added overhead begins to constrain heap space or trigger more frequent garbage collection.
What is the approximate per‑topic memory footprint when Auto‑Topic Creation is active? At what count of concurrently auto‑created topics does broker heap usage exceed a given threshold (e.g., 70 % of the configured max heap)? Does enabling this feature noticeably increase GC pause times in a low‑traffic scenario?
32275 reputation · 25 Jun 2023, 22:44 UTC
With auto‑topic creation enabled, each newly created topic consumes roughly 200‑400 KB of broker heap (metadata, subscription cursor objects and ledger references). Assuming a typical Standalone launch with a 1 GB maximum heap, the broker reaches about 70 % heap usage (~700 MB) after approximately 1 500‑3 000 concurrently auto‑created topics in a low‑traffic workload. At that point young‑generation GC pause times tend to rise from under 10 ms to the 20‑40 ms range, while old‑gen collections remain largely unchanged.
Each auto‑created topic creates a topic metadata entry in the broker’s internal cache, a subscription cursor (even if no subscriptions are explicitly created, the broker maintains a default cursor), and a reference to the BookKeeper ledger that stores the topic’s metadata. These objects live in the broker JVM heap. Because ZooKeeper and BookKeeper share the same JVM in Standalone mode, the heap available for the broker is reduced, but the per‑topic cost stays roughly constant. With no message payload retained, the overhead is dominated by the metadata structures, leading to a linear increase in heap consumption as topics are added.
pulsarbin/pulsar standalone -jvm "-Xms1g -Xmx1g".pulsar-admin topics create while monitoring broker heap via JMX (java.lang:type=Memory) or the pulsar-admin broker stats endpoint.-Xlog:gc*) to capture pause times and compare baseline (no auto‑topic creation) against the loaded scenario.If your Standalone instance uses a different maximum heap size, simply scale the topic‑count estimate proportionally (e.g., with a 2 GB heap the 70 % threshold would be reached after roughly 3 000‑6 000 topics). Please confirm the configured max heap size for your deployment if you need a precise threshold.
Use comments to ask for clarification. Post a solution as an answer.
32,275 reputation · 26 Jun 2023, 03:20 UTC
The 200‑400 KB per‑topic figure cited in the earlier answer reflects only the broker‑side heap objects (topic metadata, cursor placeholder, and ledger reference). In Standalone mode the same JVM also hosts ZooKeeper and BookKeeper, so each auto‑created topic additionally consumes ZooKeeper znode storage (a few hundred bytes) and BookKeeper ledger metadata (off‑heap direct buffers that still count toward the JVM’s total memory). Because these components share the broker’s max heap, the effective heap available for pure broker data is lower, meaning the 70 % threshold (~700 MB) is reached with fewer topics—roughly 1 800‑2 200 topics under the same 1 GB heap setting.
Furthermore, a default subscription cursor is allocated only when a subscription is actually created (e.g., a consumer connects). If no subscriptions exist, the broker keeps a lightweight placeholder, reducing the per‑topic cost slightly. This distinction explains why GC pause times begin to rise noticeably after ~2 000 topics rather than the higher counts sometimes quoted.