Managing Resource Contention in Pulsar with Multi-Tenancy and Isolation

The Noisy Neighbor Problem in Shared Messaging

When multiple teams share a single messaging cluster, a single runaway producer or a stalled consumer can degrade performance for everyone. In traditional messaging systems, you often face a binary choice: deploy a dedicated cluster for every team (which is an operational nightmare) or put everyone in one global namespace and hope they don't crash the system.

Apache Pulsar solves this through a native hierarchical isolation model. By structuring data as Tenant > Namespace > Topic, you can enforce administrative boundaries and resource limits without the overhead of managing multiple physical clusters.

The Hierarchy of Isolation

Pulsar's multi-tenancy isn't just a naming convention; it is a functional boundary for security and resource management.

• Tenants (Properties): The highest level of isolation. A tenant represents an organization or a business unit. Authentication and authorization policies are typically applied here, ensuring Team A cannot access Team B's data.
• Namespaces: These are logical groupings within a tenant. This is where the actual engineering decisions happen. You define retention policies, Time-to-Live (TTL), and backlog quotas at the namespace level, which then apply to all topics within that namespace.
• Topics: The individual channels for messages. While topics are the unit of data, they inherit the governance of their parent namespace.

Moving from Logical to Physical Isolation

Logical isolation (namespaces) prevents data leakage and manages quotas, but it doesn't stop a high-throughput namespace from consuming all the CPU on a specific Broker. To solve this, Pulsar uses Isolation Groups.

Isolation groups allow you to pin specific namespaces to a subset of brokers. If you have a "Critical-Payment-Service" namespace and a "Dev-Testing" namespace, you can ensure the payment service runs on dedicated hardware, preventing a developer's load test from causing latency spikes in production payments.

Worked Example: Provisioning an Isolated Environment

To implement this, you use the pulsar-admin CLI. This example assumes you have a running cluster and administrative permissions.

1. Create the Tenant:
Run this on the admin host to establish the top-level boundary.

bin/pulsar-admin tenants create engineering-dept

2. Create the Namespace:
Create a namespace for a specific project within that tenant.

bin/pulsar-admin namespaces create engineering-dept/payment-api

3. Set a Retention Policy:
To prevent the storage layer (BookKeeper) from filling up, set a retention policy that keeps messages for 24 hours or up to 10GB, whichever comes first.

bin/pulsar-admin namespaces set-retention engineering-dept/payment-api --size 10G --time 24h

Verification: You can verify the policy is active by running bin/pulsar-admin namespaces get-retention engineering-dept/payment-api. Any topic created within this namespace will now automatically adhere to these limits.

Trade-offs and Limitations

While multi-tenancy is powerful, it introduces a metadata tax. Every tenant and namespace creates entries in ZooKeeper. If you create thousands of namespaces for granular isolation, you may experience increased metadata latency and slower broker startup times.

Additionally, physical isolation via isolation groups is not "automatic." It requires manual mapping of namespaces to brokers and careful capacity planning. If you pin a namespace to a broker that becomes overloaded, Pulsar will not automatically move it to a less-loaded broker unless you reconfigure the isolation group.

Actionable Summary

To avoid resource exhaustion in a shared Pulsar cluster: first, map your organizational structure to Tenants. Second, group related services into Namespaces to centralize retention and TTL settings. Finally, use Isolation Groups only for your most critical workloads to protect them from the "noisy neighbor" effect of lower-priority environments.