Isolating Environments with Apache Pulsar Multi-tenancy: Tenants, Namespaces, and Practical Policies

The Isolation Dilemma

Teams often face a choice when scaling a messaging platform: run a separate Pulsar cluster for each team or environment to guarantee isolation, or share a single cluster and risk one noisy workload affecting everyone else. Managing many clusters adds operational overhead, while a shared cluster can suffer from resource contention if policies are not enforced.

Apache Pulsar offers a hierarchical multi-tenancy model that lets you isolate policies and permissions without provisioning extra clusters. This post shows how to use tenants and namespaces to separate environments such as production and staging, and highlights the limits of logical isolation.

Tenants and Namespaces in Pulsar

Pulsar organizes resources in a three‑level hierarchy:

1. Tenant (Property) – the top‑level administrative boundary. Authentication and authorization are configured here; a tenant typically maps to a business unit, team, or external client.
2. Namespace – lives inside a tenant and is the unit for applying operational policies such as message retention, TTL, or backlog quotas. All topics under a namespace inherit these settings.
3. Topic – the actual message stream; it belongs to exactly one namespace.

Because the hierarchy is stored in ZooKeeper, changes to tenants or namespaces are reflected cluster‑wide without restarting brokers or bookies.

Worked Example: Separating Production and Staging

The following commands assume you have access to a Pulsar cluster with admin privileges (e.g., the superuser role or a token with tenant‑creation permissions). Adjust the tenant name to suit your organization.

# 1. Create a tenant for the engineering organization
pulsar-admin tenants create engineering

# 2. Create namespaces for different environments
pulsar-admin namespaces create engineering/production
pulsar-admin namespaces create engineering/staging

# 3. Apply a retention policy only to the staging namespace
#    Keep messages for 24 hours or up to 10 GB, whichever comes first.
pulsar-admin namespaces set-retention engineering/staging --size 10G --time 24h

# 4. Verify the policy on staging
pulsar-admin namespaces get-retention engineering/staging

# 5. Verify that production inherits the cluster default (no custom retention)
pulsar-admin namespaces get-retention engineering/production

When you run the verification commands, the output for engineering/staging should show the 24 hour/10 GB settings you just applied, while engineering/production will display the cluster‑wide defaults (or whatever retention is configured at the namespace level). This demonstrates that policies set on a namespace do not leak to sibling namespaces.

Trade‑off: Logical Isolation Only

Pulsar’s multi-tenancy provides logical separation of access control and policy enforcement. Compute (brokers) and storage (bookies) resources are still shared across all tenants and namespaces unless you explicitly limit usage.

If a producer in the engineering/staging namespace publishes at a rate that saturates a broker’s CPU or network interface, traffic in engineering/production may experience increased latency or throttling. To mitigate this, you can enforce dispatch rates or backlog quotas at the namespace level:

# Limit the maximum unacknowledged messages for the staging namespace
pulsar-admin namespaces set-backlog-quota engineering/staging --limit 5GB --policy producer_request_hold

# Optionally set a dispatch rate (messages per second) per producer
pulsar-admin namespaces set-dispatch-rate engineering/staging --rate 1000

These quotas are also stored in ZooKeeper and enforced by the broker.

Operational Limits to Consider

Each namespace creates a ZooKeeper node that stores its configuration. Creating a namespace for every topic can lead to noticeable metadata overhead, slowing down topic lookups and increasing cluster recovery time after a failure. A practical guideline is to group topics that share the same retention, TTL, or quota requirements into a single namespace, and to create additional namespaces only when a distinct policy set is needed.

Closing Action

If you maintain multiple Pulsar clusters solely to isolate teams or environments, review the resource usage of those clusters. In many cases you can consolidate them into a single cluster by defining tenants for each team and using namespaces to enforce environment‑specific policies. Remember to pair namespace‑level policies with backlog quotas or dispatch rates to protect against noisy‑neighbor effects, and monitor ZooKeeper metadata size as you scale the number of namespaces.