Designing Apache Pulsar Tiered Storage Offload: Requirements, Architecture, and Operational Guidance

Problem Statement

Many Pulsar deployments need to keep billions of messages for months or years without incurring the storage cost of keeping every ledger on local disk. Pulsar’s tiered storage offload feature lets you move old BookKeeper ledgers to an external object store (S3, GCS, Azure Blob) while keeping the same producer/consumer API. The key question is: how do you design the cluster, secure the data path, monitor the pipeline, and know when the design should be changed?

Requirements

• Cluster with at least one broker and a BookKeeper ensemble.
• Supported Pulsar version 3.0+ (offload is stable in 3.3+).
• Object store bucket with sufficient capacity and the correct lifecycle policy.
• IAM or key‑based credentials that allow the broker to PUT and GET objects.
• TLS enabled for all broker‑object‑store traffic.
• Monitoring stack (Prometheus/ Grafana) to expose pulsar_broker_offload_* metrics.

Minimal Suitable Design

For most workloads a single Pulsar cluster is enough. The design relies on per‑namespace policy flags that enable offload for ledgers older than a configurable threshold. The broker serves recent messages from BookKeeper; when a consumer seeks a message older than the threshold, the broker transparently fetches the ledger from the object store.

Cluster Topology

Broker 1  -----------  BookKeeper 1
   |                         |
   |                         |
   |                         |
Broker 2  -----------  BookKeeper 2
   |                         |
   |                         |
   |                         |
   +-----------+--------------
               |  Object Store
               |  (S3/GCS/Azure)

All brokers are configured with the same object‑store credentials. The object store is not part of the Pulsar cluster; it is an external trusted service.

Namespace Policy Example

pulsar-admin namespaces set-txn-conf my-tenant/my-namespace \
    --tieredStorageEnabled true \
    --offloadThreshold 30d \
    --offloadDriver s3 \
    --offloadS3Bucket mybucket \
    --offloadS3Region us-east-1 \
    --offloadS3AccessKey YOUR_ACCESS_KEY \
    --offloadS3SecretKey YOUR_SECRET_KEY

Replace the placeholders with your bucket name, region, and credentials. The --offloadThreshold value is the age after which a ledger is considered “cold” and eligible for offload.

Trust and Data Boundaries

The critical trust boundary is between the broker process and the object store endpoint. Brokers must authenticate with the store using credentials that are scoped only to the bucket used for offloading. IAM policies should enforce:

• Read/write access for the pulsar-service-account.
• No cross‑bucket access.
• Encryption at rest enabled on the bucket (S3 SSE‑S3, GCS default).

All traffic between broker and object store must be encrypted via TLS. Pulsar’s brokerServiceUrlTLS and brokerServiceUrl settings should point to the same endpoint, and the brokerServiceUrlTLS must be used for offload communication.

Operational Checks

Monitoring is essential to confirm that offload jobs complete and that consumers can read offloaded data. The following Prometheus metrics are the most useful:

Metric	Description
pulsar_broker_offload_lag_seconds	Time since a ledger became eligible for offload.
pulsar_broker_cold_read_latency_seconds	Latency of reads that hit the object store.
pulsar_broker_offload_errors_total	Number of failed offload attempts.
pulsar_broker_offload_success_total	Number of successful offloads.
pulsar_broker_ledger_rollovers_total	How often ledgers roll over (triggering potential offload).

Operational checks:

1. Verify Offload Lag: pulsar_broker_offload_lag_seconds should stay below a few hours for a 30‑day threshold. A growing lag indicates a bottleneck.
2. Cold‑Read Latency: Keep pulsar_broker_cold_read_latency_seconds under 200 ms for most workloads. If it spikes, investigate network or bucket performance.
3. Error Rates: Any non‑zero pulsar_broker_offload_errors_total warrants a deeper dive into bucket permissions or network stability.
4. Consumer Health: Run a test consumer that reads messages older than the threshold and confirm no errors. Log the first and last message timestamps to verify reachability.

Failure Modes and Mitigation

• Object‑Store Outage: Reads of offloaded data fail, but writes continue to BookKeeper. Consumers will see a MessageNotFoundException if the ledger is missing. Mitigation: implement a fallback retry policy and alert on cold_read_latency spikes.
• Permission Revocation: If IAM credentials lose write permission, offload jobs abort. The broker logs will emit AccessDeniedException. Mitigation: monitor offload_errors and set up automated credential rotation.
• Data Corruption or Deletion: Offloaded segments can become unreadable if the bucket is accidentally cleaned. Pulsar does not automatically recover deleted objects. Mitigation: enable bucket versioning or immutable storage tiers.
• Immediate Offload on Enable: Enabling tiered storage on a namespace with a large history triggers an immediate offload of all existing ledgers. This can saturate the object store and network. Mitigation: pre‑stage the bucket with enough space and throttle the offload job via offloadThrottle.

When to Redesign

Keep the current design if:

• Retention period > 7 days and cost savings outweigh the cold‑read latency.
• Compliance allows data to reside in an external store and encryption at rest is acceptable.
• Read patterns are mostly “hot” and occasional cold reads are tolerable.

Consider redesigning if:

• Retention < 7 days: keeping data on local disk is cheaper and simpler.
• Regulatory constraints forbid external storage or require immediate access to all data.
• Cold‑read latency must stay under 50 ms for latency‑sensitive consumers.
• Operational complexity (credentials, monitoring, bucket capacity) becomes high relative to the benefit.

Practical Verification Steps

1. Create a test namespace with tiered storage enabled and a 1‑day threshold.
2. Produce a burst of messages to force a ledger rollover.
3. Wait until the ledger age exceeds the threshold and check offload_success_total in Prometheus.
4. Start a consumer that reads from the beginning of the topic; verify that the first messages are retrieved from the object store (check broker logs for ColdRead entries).
5. Simulate an object‑store outage by revoking the bucket policy; confirm that new writes succeed but cold reads fail with a clear error.

Use the pulsar-admin topics stats command to confirm that the namespace still reports the correct number of retained messages. If the statistics show a sudden drop, investigate offload failures.

Conclusion

Tiered storage offload in Pulsar is a powerful tool for long‑term data retention, but it introduces new trust boundaries and operational responsibilities. By following the minimal design outlined above, securing the broker‑object‑store channel, and monitoring the key metrics, you can keep the system healthy while reducing storage costs. Always validate the design in a staging environment and be ready to pivot if compliance or latency constraints change.