Pulsar maxConcurrentLookupRequest without per-tenant isolation for concurrent lookup latency
0 reputation · 11 May 2025, 20:22 UTC
We run a multi-tenant Apache Pulsar cluster (assume a recent 3.x broker; behavior may differ on older releases) and see lookup latency spikes that only appear when many clients issue topic lookups concurrently. Produce and consume latency stay flat, so the bottleneck appears to be broker-side lookup handling rather than BookKeeper or ZooKeeper throughput.
The broker setting maxConcurrentLookupRequest caps simultaneous lookups, but as documented it is a single global limit per broker. There is no documented way to subdivide it by tenant, namespace, or role, so one tenant doing a mass consumer reconnect can saturate the lookup queue and inflate tail latency for everyone else.
Raising the limit risks overloading the shared handler threads and the ZooKeeper/metadata path; lowering it turns the noisy-neighbor problem into outright queuing or rejection for all tenants. Neither option gives a permission or quota boundary tied to authentication.
Is there any supported mechanism (broker config, namespace policy, or proxy-level control) to enforce fair or per-tenant lookup concurrency rather than one global cap?
If not, what is the expected broker behavior when the limit is hit — immediate rejection, queuing, or timeout — and is it consistent across recent versions?
Which broker metrics best confirm that lookup concurrency saturation, not ZooKeeper latency, is the cause of the tail latency?