Designing Multi-tenant Isolation in Apache Pulsar

The Multi-tenancy Challenge: Preventing Noisy Neighbors

In a shared messaging infrastructure, a single high-volume producer can saturate broker CPU or fill storage disks, causing latency spikes for every other application on the cluster. This "noisy neighbor" effect is the primary risk when consolidating multiple teams or clients onto a single Apache Pulsar cluster.

The solution is to implement a strict hierarchical isolation strategy. By leveraging Pulsar's native multi-tenancy, you can shift from a flat topic structure to a tiered model that enforces administrative and resource boundaries at the point of entry.

The Minimum Viable Isolation Architecture

To achieve isolation without over-engineering, Pulsar uses a three-tier hierarchy: Property (Tenant) > Namespace > Topic. This structure allows you to apply policies at the broadest level that trickle down to individual streams.

1. The Property (Tenant) Layer

The Property is the highest level of isolation. It is primarily an administrative boundary. All authentication and authorization tokens are mapped to a property, ensuring that a user from "Tenant-A" cannot even see the existence of topics belonging to "Tenant-B".

2. The Namespace Layer

Namespaces are the primary unit of operational control. This is where you define the "how" of your data handling. Key configurations applied here include:

• Retention Policies: How long messages are kept after consumption.
• TTL (Time to Live): When unacknowledged messages are automatically discarded.
• Backlog Quotas: Limits on how much data can accumulate before the producer is throttled.

3. The Topic Layer

Topics are the actual data channels. While policies are set at the namespace level, topics are the unit of distribution across the broker fleet.

Trust and Data Boundaries

Data isolation in Pulsar is not automatic; it relies on the integration of the Authentication Provider and the Authorization Plugin. When a client connects, the broker validates the token and checks if the user has the required role (e.g., producer or consumer) for the specific property and namespace requested.

Because Pulsar separates the serving layer (Brokers) from the storage layer (Bookies), the data boundary extends to the disk. While Bookies store data for multiple tenants in the same ledgers, the Broker acts as the gatekeeper, ensuring no client can request a ledger fragment they are not authorized to access.

Operational Implementation and Verification

To implement this, you must use the pulsar-admin CLI tool. These commands should be run from a management node with network access to the Pulsar broker and ZooKeeper.

Step 1: Establish the Tenant and Namespace

# Create a tenant (Property)
pulsar-admin tenants create my-tenant

# Create a namespace within that tenant
pulsar-admin namespaces create my-tenant/production-ns

Step 2: Apply Resource Quotas

To prevent a single namespace from consuming all available disk space, apply a storage quota. This is a critical check to prevent cluster-wide outages.

# Limit the namespace to 10GB of backlog storage
pulsar-admin namespaces set-backlog-quota my-tenant/production-ns 10G

Step 3: Verification of Isolation

To verify the boundary is working, attempt to produce a message to a tenant you are not authorized for using a client configured with a different token. The expected result is a 403 Forbidden or AuthorizationException. To verify the quota, produce data until the 10GB limit is reached; the producer should experience increased latency or receive a ProducerQueueFullException as the broker throttles the input.

Failure Modes and Design Shifts

Failure Mode	Impact	Mitigation
ZooKeeper Metadata Bloat	Slow topic creation/lookup	Avoid creating thousands of namespaces; use a few large namespaces with many topics.
Bookie Disk Hotspot	I/O bottlenecks for specific tenants	Increase the ensemble size (number of Bookies involved in a ledger) to spread the load.
Broker CPU Saturation	Latency for all tenants on that broker	Implement Broker-level resource isolation via Kubernetes pods or separate clusters for high-priority tenants.

When to Change This Design

The hierarchical model works for most organizational needs. However, you should move to a Physically Isolated Cluster (separate brokers and bookies) if:

• You have strict regulatory requirements (e.g., PCI-DSS) that forbid data from different clients residing on the same physical disk.
• A single tenant's throughput exceeds the total network bandwidth of the broker nodes.
• You require custom BookKeeper configurations (like different journal disk types) for one specific tenant.