Solving the Noisy Neighbor Problem with Pulsar Multi-tenancy

The Shared Cluster Dilemma

When multiple teams share a single messaging cluster, a single runaway producer or an inefficient consumer group can degrade performance for everyone. In traditional messaging systems, you often face a binary choice: manage a sprawling fleet of small, isolated clusters or risk a "noisy neighbor" crashing your production pipeline.

Apache Pulsar solves this through a hierarchical multi-tenancy model. Instead of treating the cluster as a flat pool of topics, Pulsar organizes data into Properties (Tenants), Namespaces, and Topics. This allows you to apply administrative boundaries and resource constraints without the overhead of deploying separate infrastructure for every team.

The Hierarchy: Tenants vs. Namespaces

To use multi-tenancy effectively, you must distinguish between the administrative boundary (Tenant) and the operational boundary (Namespace).

• Tenants: These are the highest level of isolation. A tenant is typically mapped to a business unit or a department. Authentication and authorization policies are defined here. If you are providing "Messaging as a Service" internally, each requesting team gets their own tenant.
• Namespaces: These sit inside tenants. Namespaces are where you define the actual behavior of the data. This includes retention policies (how long data is kept after consumption), TTL (Time to Live for unconsumed messages), and backlog quotas.

Because Pulsar separates the serving layer (Brokers) from the storage layer (BookKeeper), you can scale the storage for a high-volume tenant without needing to add more brokers for the entire cluster.

Implementing Logical Isolation

Isolation in Pulsar can be "soft" (logical separation via ACLs) or "hard" (dedicated broker groups). For most engineering teams, logical isolation via the pulsar-admin CLI is the starting point.

To implement this, you first establish the tenant, then the namespace, and finally the topic. This structure ensures that a developer in team-alpha cannot accidentally produce messages to a topic owned by team-beta.

Worked Example: Configuring a Restricted Namespace

In this scenario, we create a tenant for the "Payments" team and a namespace for "Audit Logs" that has a strict 7-day retention policy to meet compliance requirements without affecting other team logs.

Run these commands on a machine with pulsar-admin installed and network access to the Pulsar manager URL (default port 8080). You will need administrative privileges to create tenants.

# 1. Create the tenant for the Payments team
pulsar-admin tenants create payments-team

# 2. Create a namespace within that tenant
pulsar-admin namespaces create payments-team/audit-logs

# 3. Set a retention policy: Keep messages for 7 days, regardless of consumption
pulsar-admin namespaces set-retention payments-team/audit-logs --size 10GB --time 7d

# 4. Create a topic within that namespace
pulsar-admin topics create persistent://payments-team/audit-logs/transaction-history

Verification: To verify the isolation, attempt to produce a message to this topic using a client authenticated as a different tenant. The broker should return an AuthorizationException. You can also check the retention settings by running pulsar-admin namespaces get-retention payments-team/audit-logs.

Trade-offs and Metadata Overhead

While multi-tenancy is powerful, it is not free. Every namespace and topic creates metadata that must be stored in the configuration store (typically ZooKeeper or etcd). Over-provisioning thousands of tiny namespaces can lead to increased metadata pressure, slowing down broker startup times and topic lookups.

Additionally, while logical isolation prevents unauthorized access, it does not automatically prevent resource exhaustion. If one tenant floods the cluster with messages, they can still saturate the BookKeeper ledger disks. To prevent this, you must implement Resource Quotas at the tenant level to cap the total storage or throughput a single entity can consume.

Practical Summary for Architects

When designing your Pulsar topology, avoid the temptation to create a 1:1 mapping between topics and namespaces. Instead, group topics by their lifecycle and compliance needs. If five topics all require a 30-day retention period and the same access permissions, they belong in one namespace. If one topic requires 7-year archiving for legal reasons, it needs its own namespace.