Architecting Multi-Tenant Isolation in Apache Pulsar

The Challenge of Shared Messaging Infrastructure

When deploying a messaging system for multiple teams or external clients, the primary risk is noisy neighbor interference and unauthorized data access. Without strict isolation, a single producer flooding a topic can exhaust broker memory or disk I/O, impacting every other user on the cluster. The goal is to provide a shared physical infrastructure that behaves like dedicated virtual instances.

The Hierarchical Isolation Model

Pulsar solves this through a three‑tier administrative hierarchy: Property (Tenant) > Namespace > Topic. This structure defines where security policies are applied and how resources are partitioned.

• Tenant (Property): The highest level of isolation. Tenants are used to group namespaces and are typically mapped to a business unit or a customer. Authentication and administrative permissions are usually managed at this boundary.
• Namespace: A logical grouping of topics. This is the primary unit for operational configuration. You apply retention policies, quotas, and authorization rules here rather than on individual topics.
• Topic: The actual stream of messages. Topics inherit the policies of their parent namespace.

Smallest Suitable Design for Isolation

To achieve basic multi‑tenancy without over‑engineering, implement a design that separates the Serving Layer (Brokers) from the Storage Layer (BookKeeper). Because Brokers are stateless, they do not store the data they serve; they merely coordinate the flow between producers and Bookies (storage nodes).

Trust and Data Boundaries

1. Authentication: Use JSON Web Tokens (JWT) or TLS certificates. The broker validates the token against a trusted provider before allowing any connection.
2. Authorization: Policies are applied at the namespace level. For example, you can grant produce permissions to Team A for tenant1/namespace-a while denying them access to tenant1/namespace-b.

Operational Configuration Example

To isolate two teams (Team Alpha and Team Beta), you must define the tenants and namespaces via the pulsar-admin CLI. Run these commands from a terminal with access to the Pulsar admin API.

# 1. Create Tenants
bin/pulsar-admin tenants create team-alpha
bin/pulsar-admin tenants create team-beta

# 2. Create Namespaces within those tenants
bin/pulsar-admin namespaces create team-alpha/production
bin/pulsar-admin namespaces create team-beta/production

# 3. Set a message size quota for Team Beta to prevent resource exhaustion
# This limits the total size of messages in the namespace to 10GB
bin/pulsar-admin namespaces set-backlog-quota team-beta/production 10GB

Risk: Applying overly restrictive quotas can cause producers to block or receive errors when the limit is reached. Always monitor the pulsar_storage_backlog_quota_exceeded metric to determine if limits need adjustment.

Failure Modes and Recovery

The decoupled architecture ensures that failures in one layer do not necessarily crash the entire tenant environment.

Failure Scenario	Impact	Mitigation
Broker Crash	Temporary loss of connection for topics owned by that broker.	Other brokers automatically take ownership of the orphaned topics via ZooKeeper coordination.
Bookie Failure	Potential loss of a data fragment.	Ensemble replication ensures data is mirrored across multiple Bookies; the system automatically replicates missing fragments.
Metadata Store Latency	Slow topic creation or policy updates.	Avoid high churn (rapidly creating/deleting topics), as this stresses ZooKeeper.

When to Evolve the Design

The single‑cluster multi‑tenant model works until your requirements shift from logical isolation to geographic or regulatory isolation. You should move to a Geo‑Replicated design if:

• Data Sovereignty: Laws require that data for a specific tenant remains within a specific country’s borders.
• Disaster Recovery: You require a zero‑RPO (Recovery Point Objective) failover where a secondary cluster takes over instantly if the primary region fails.
• Latency: Producers and consumers are globally distributed, and the round‑trip time to a single central cluster is unacceptable.

Verification Step

To verify isolation, create a user with a JWT token assigned only to team-alpha. Attempt to produce a message to team-beta/production/my-topic. The broker should return an AuthorizationException, confirming that the namespace boundary is functioning.