Using Apache Pulsar Tiered Storage to Cut Costs on Long‑Term Message Retention

The problem: retaining messages without blowing up storage bills

Many teams need to keep Pulsar topics for weeks or months to satisfy audit or replay requirements. As the volume grows, the BookKeeper ledgers that store message segments consume expensive SSD or NVMe space, while older data is rarely accessed. Scaling the cluster just to hold cold data quickly becomes cost‑ineffective.

Thesis: tiered storage lets you keep recent data hot and move older segments to cheap object storage

Apache Pulsar’s tiered storage feature offloads sealed BookKeeper ledger segments to an external bucket (S3, GCS, Azure Blob, etc.) via an asynchronous offloader service that runs alongside each broker. Recent segments stay on local disks for low‑latency reads; when a consumer asks for a segment that has been offloaded, the broker transparently fetches it from the object store, preserving ordering and exactly‑once guarantees.

How tiered storage works

• Sealing: A ledger segment is sealed when it reaches the configured rollover size (default 10 MB) or time.
• Offload: The offloader service copies the sealed segment to the bucket defined in the namespace policy.
• Fetch on demand: If a consumer requests data from an offloaded segment, the broker issues a read request to the object store, streams the data back, and serves it to the client.

Because the offloader runs asynchronously, enabling or disabling tiered storage does not require a broker restart.

Enabling tiered storage – a concrete example

Below is a minimal, reproducible setup for a test cluster running Pulsar 2.9 or later. All commands assume you have access to the broker host and a Pulsar admin CLI.

1. Broker configuration

# in broker.conf (add or edit)
tieredStorageEnabled=true
# offload driver for S3; replace with gcs or azure if needed
tieredStorageOffloadDriver=s3
# optional: tune offloader threads (default is number of cores)
tieredStorageOffloaderNumThreads=4

After saving, reload the configuration:

bin/pulsar-admin brokers update-config --config tieredStorageEnabled=true
bin/pulsar-admin brokers update-config --config tieredStorageOffloadDriver=s3

2. Namespace policy

Define where and when segments should be offloaded. In this example we offload after 5 MB or 1 hour, whichever comes first, to a test S3 bucket.

bin/pulsar-admin namespaces set-offload-policy \
  --threshold 5MB \
  --age 1h \
  --destination s3://my-test-bucket/pulsar-offload/ \
  public/default

The bucket must exist and the broker’s IAM role (or AWS credentials file) must allow s3:PutObject, s3:GetObject, and s3:ListBucket on that prefix.

3. Verify offload is happening

1. Produce a steady stream of messages to a topic in the namespace, e.g., public/default/test-topic.
2. Wait for the ledger to roll over (you can watch bin/pulsar-admin topics stats for sizeInKB approaching the threshold).
3. Check the bucket:

aws s3 ls s3://my-test-bucket/pulsar-offload/ --recursive

You should see objects named like segment-00000000000000000000.gz.

Consume from the topic with a subscription:

bin/pulsar-client consume public/default/test-topic \
  --subscription-name my-sub \
  --num-messages 1000

Watch the broker logs (typically logs/pulsar.log) for lines such as:

INFO  OffloadReader - Reading offloaded segment s3://my-test-bucket/pulsar-offload/segment-00000000000000000000.gz

This confirms that the broker fetched the segment from object storage on demand.

Trade‑offs and limitations

• Read latency: Fetching an offloaded segment adds network round‑trip time and object‑store latency (typically tens to hundreds of milliseconds). Design consumers to tolerate occasional higher latency, or create separate subscriptions for hot vs. cold data.
• Egress cost: Each read incurs data transfer charges from the cloud provider. Estimate the volume of historic reads you expect and compare it to storage savings.
• Bucket lifecycle rules: If the bucket is configured to delete objects after a short period, Pulsar may lose segments it still needs. Ensure the retention period on the bucket is at least as long as the maximum message retention you configure in Pulsar.
• Operational overhead: You must monitor offload lag, offloader throughput, and bucket health. Pulsar exposes metrics like tieredStorageOffloadLag and tieredStorageFetchLatency via Prometheus.

Actionable closing

Start small: enable tiered storage on a non‑critical namespace, point it to a test bucket, and set a low offload threshold (e.g., 1 MB) to observe the behavior quickly. Verify that sealed segments appear in the bucket and that reads trigger fetches as shown in the logs. Once you’re comfortable, adjust the threshold to match your production retention window, monitor cost and latency, and roll out to additional namespaces. Tiered storage isn’t a silver bullet, but when configured with an eye on access patterns and egress costs, it can dramatically reduce the expense of keeping months of Pulsar messages available for replay or audit.