Cutting Pulsar Storage Costs with Tiered Storage: A Practical Guide

The Problem: BookKeeper Costs at Scale

You're running a Pulsar cluster for event sourcing and audit logs. Retention is set to 90 days for compliance. BookKeeper stores everything on fast disks, and the bill is growing 60% quarter over quarter. The data is rarely read after the first week, but you can't delete it. You need a way to move older segments to cheaper storage without rewriting consumers or losing ordering guarantees.

Thesis: Tiered Storage Moves Cold Data Without Breaking Semantics

Apache Pulsar's tiered storage (available since 2.10) offloads sealed BookKeeper segments to object storage (S3, GCS, Azure Blob) while keeping the same topic APIs. Producers and consumers see no change. The broker redirects reads to the appropriate tier. For workloads with predictable retention—audit logs, event sourcing, clickstreams—this cuts storage spend by 60–80% with a modest read-latency trade-off for cold data.

How It Works: Hot on BookKeeper, Cold in Object Storage

BookKeeper holds recent segments (hot data). When a segment is sealed—either because it reached segmentMaxSize or the broker rolled it—Pulsar can asynchronously upload it to the configured object store. The offload policy lives at the namespace or topic level. Two common triggers:

• Time-based: offload segments older than offloadDeletionLagMs (e.g., 24 hours).
• Size-based: offload when accumulated offloadable data exceeds offloadThreshold (e.g., 1 GB).

The broker coordinates with BookKeeper to seal and upload without blocking producers. Readers fetch offloaded segments via the same client APIs; the broker streams from object storage on demand. Ordering and exactly-once semantics are preserved because segments are immutable once sealed.

Worked Example: Enable Tiered Storage for an Audit Log Topic

Assume a Pulsar 2.11 cluster on Kubernetes with broker config access and an S3 bucket s3://pulsar-offload-prod. You have admin permissions on the cluster.

1. Configure the Broker

Edit broker.conf on each broker (or via your config management):

# Enable tiered storage
tieredStorageEnabled=true
# S3 backend
offloadersDirectory=./offloaders
tieredStorageDriver=org.apache.pulsar.tieredstorage.offloaders.S3Offloader
# S3 bucket and region (use IAM roles or env vars for credentials)
s3ManagedLedgerOffloadBucket=pulsar-offload-prod
s3ManagedLedgerOffloadRegion=us-east-1
# Optional: max block size for reads from S3
s3ManagedLedgerOffloadReadBufferSize=1048576

Restart brokers rolling to pick up changes.

2. Set an Offload Policy on the Namespace

Run on a machine with pulsar-admin and cluster-admin role:

pulsar-admin namespaces set-offload-policies \
  my-tenant/audit-namespace \
  --offload-threshold 1G \
  --offload-deletion-lag-ms 86400000 \
  --offload-max-block-size 64MB

This tells Pulsar: start offloading when 1 GB of sealed segments accumulate, or after 24 hours, whichever comes first.

3. Produce Test Data and Verify Offload

# Produce ~2 GB to a topic in that namespace
for i in {1..2000000}; do
  echo "audit-event-$i" | pulsar-client produce \
    persistent://my-tenant/audit-namespace/audit-log \
    --messages 1
  # throttle to avoid overwhelming the demo
  sleep 0.001
done

Wait a few minutes for segments to seal and offload. Then check status:

pulsar-admin topics offload-status \
  persistent://my-tenant/audit-namespace/audit-log

Expected output shows Offload completed with byte counts. If you see Offload in progress, wait and re-run.

4. Verify Data Integrity

Consume from the earliest message and compare checksums:

pulsar-client consume \
  persistent://my-tenant/audit-namespace/audit-log \
  --subscription-name verify-offload \
  --subscription-type Exclusive \
  --num-messages 0 \
  --receiver-queue-size 1000 \
  --print-message-body false | \
  sha256sum

Compare with a checksum of the original payload. They should match.

Trade-offs and Limitations

Factor	Impact
Read latency for cold data	Adds one S3 round-trip per segment (typically 10–50 ms). Keep latency-sensitive topics fully on BookKeeper.
Immutable offloaded segments	Compaction and deduplication only apply to data still on BookKeeper. Plan retention so compaction runs before offload if you need it.
Egress and request costs	Frequent replay of large offloaded datasets incurs S3 egress and GET charges. Estimate access patterns; consider a separate replay cluster.
Geo-replication	Each cluster needs its own offload config. Offloaded data is not replicated automatically.
Upgrade risk	Offload metadata formats can change between Pulsar versions. Test upgrades in staging with a representative offloaded dataset.

Monitoring: Keep Offload Pace with Ingestion

Two signals matter:

• pulsar_tiered_storage_offload_bytes_total (counter) — cumulative bytes offloaded. Should grow steadily.
• pulsar_tiered_storage_offload_latency_seconds (histogram) — time from segment seal to upload completion. Spikes indicate S3 throttling or network issues.

Alert if offload lag (bytes pending) exceeds your threshold for more than 15 minutes. The offload-status command also shows pending bytes per topic.

Actionable Next Steps

1. Identify namespaces with long retention and low cold-read frequency (audit logs, event stores).
2. Provision an object store bucket with lifecycle policies (e.g., transition to Glacier after 30 days).
3. Enable tiered storage in a staging cluster. Run the verification steps above with a representative dataset.
4. Set conservative offload thresholds (1 GB / 24 h) and monitor for two weeks before tightening.
5. Document the upgrade test procedure: snapshot offloaded metadata, upgrade, verify reads.

Tiered storage is a low-risk, high-leverage knob for Pulsar operators. The feature is mature, the semantics are preserved, and the cost savings are real—provided you respect the read-latency and egress trade-offs.