Pulsar maxConcurrentLookupRequest without per-tenant isolation for concurrent lookup latency

Question

We run a multi-tenant Apache Pulsar cluster (assume a recent 3.x broker; behavior may differ on older releases) and see lookup latency spikes that only appear when many clients issue topic lookups concurrently. Produce and consume latency stay flat, so the bottleneck appears to be broker-side lookup handling rather than BookKeeper or ZooKeeper throughput.

The broker setting maxConcurrentLookupRequest caps simultaneous lookups, but as documented it is a single global limit per broker. There is no documented way to subdivide it by tenant, namespace, or role, so one tenant doing a mass consumer reconnect can saturate the lookup queue and inflate tail latency for everyone else.

Raising the limit risks overloading the shared handler threads and the ZooKeeper/metadata path; lowering it turns the noisy-neighbor problem into outright queuing or rejection for all tenants. Neither option gives a permission or quota boundary tied to authentication.

Is there any supported mechanism (broker config, namespace policy, or proxy-level control) to enforce fair or per-tenant lookup concurrency rather than one global cap?

If not, what is the expected broker behavior when the limit is hit — immediate rejection, queuing, or timeout — and is it consistent across recent versions?

Which broker metrics best confirm that lookup concurrency saturation, not ZooKeeper latency, is the cause of the tail latency?

Tasadduq Burney · Answer

There is no supported mechanism to subdivide maxConcurrentLookupRequest by tenant, namespace, or role in Pulsar 3.x brokers. The setting is a single broker-global semaphore that guards the lookup path; namespace policies (resource groups, rate limits) control message traffic, not lookup concurrency.

When the cap is hit, the broker uses a bounded semaphore on the lookup path. Additional lookup requests fail fast with a TooManyRequests-style rejection rather than being queued indefinitely. The client then retries per its lookup/retry policy, which is why a mass reconnect by one tenant inflates tail latency for all tenants sharing that broker.

Likely explanation (not confirmed for your exact build): lookup-permit exhaustion on the brokers owning hot topic bundles, amplified by client retry storms. The fact that produce and consume latency stay flat is consistent with this, since the message path uses separate rate limiters and thread pools.

Practical mitigations without per-tenant quotas:

Spread topic bundles across more brokers (increase bundle count, enable load shedding) to reduce per-broker lookup volume.
Front lookups with the Pulsar proxy so the concurrency cap applies per-proxy instance rather than a single broker.
Stagger client reconnects using exponential backoff and jitter to avoid simultaneous permit exhaustion.
Reduce lookup frequency via partitioned-topic-aware clients and connection reuse.

Metrics to confirm lookup-queue saturation (not ZooKeeper/metadata latency):

pulsar_broker_lookup and pulsar_broker_lookup_answers latency should rise during spikes.
Rejected/lookup throttling counters (e.g., pulsar_broker_lookup_rejections where exposed) climbing in step with spikes isolates the cause.
Broker thread-pool usage for lookup/executor pools.
ZooKeeper operation latency should stay flat if the bottleneck is broker-side concurrency.

If you can share your exact Pulsar 3.x broker version and whether clients connect directly or through a proxy, I can point at the specific metric names and config paths that apply to your build.

Pulsar maxConcurrentLookupRequest without per-tenant isolation for concurrent lookup latency

1 answer

Your perspective belongs here.

0 question comments