When Your Pulsar Cluster Outgrows Its Disks: Tiered Storage in Practice

The disk bill nobody budgeted for

You set a 30-day retention policy on a high-throughput Pulsar topic because the data team wants replay capability. Three months later, BookKeeper is holding terabytes of cold data on SSDs that cost as much as the compute serving them, and 95% of reads hit only the last few hours of the log. This is the exact scenario Apache Pulsar's tiered storage feature exists for: it automatically moves older, sealed segments from BookKeeper to cheap object storage (S3, GCS, Azure Blob, or even a local filesystem) while consumers keep reading as if nothing changed.

The useful takeaway: tiered storage is a policy decision, not a migration. You configure a threshold, Pulsar offloads asynchronously, and the read path stays transparent. But the details of when offload triggers and how it interacts with retention and geo-replication determine whether it saves money or creates operational surprises.

How offload actually works

BookKeeper writes each topic's data as a sequence of immutable segments. Once a segment is sealed (rolled over) and the backlog exceeds your configured threshold, the broker's offloader copies that segment to the object store and deletes the local copy, leaving only metadata behind. Consumers reading recent data never touch the object store; consumers replaying history are served by the broker streaming objects back in original segment order. No client changes are required.

Two implications matter operationally. First, offload is segment-granular, so very recent data always stays local — you can't offload the active segment. Second, offload is asynchronous and best-effort. A broker crash mid-offload can leave orphaned objects in the bucket; Pulsar 2.10+ added idempotent retries, but occasional manual cleanup may still be needed.

Configuring it on a namespace

Policies are set per-namespace (with topic-level overrides available). Run this against your cluster's admin endpoint — it requires namespace admin permissions:

pulsar-admin namespaces set-offload-policies my-tenant/my-ns \
  --driver aws-s3 \
  --bucket my-pulsar-offload \
  --region us-east-1 \
  --offload-threshold-size 10GB \
  --max-block-size 64MB \
  --read-buffer-size 1MB

The S3 driver uses Hadoop's S3A client under the hood, so credentials, IAM instance roles, and VPC endpoints follow standard S3A conventions rather than Pulsar-specific config. The threshold can be size-based (as above) or time-based. A size threshold of 10GB means: once the local backlog exceeds 10GB, start offloading the oldest sealed segments until you're back under it.

To verify it worked, produce enough data to cross the threshold, then check:

pulsar-admin topics stats-internal persistent://my-tenant/my-ns/my-topic

Look for offloadedBytes and offloadedMessages growing, and confirm the bucket contains segment objects. Then consume from the earliest position and confirm no gaps. Expect roughly 10–50ms of added latency per batch when reading from S3 versus local disk — fine for replay workloads, noticeable for latency-sensitive ones.

Tuning and monitoring

Three knobs do most of the work. maxBlockSize (default 64MB) controls object size in the bucket: larger blocks mean fewer objects and lower per-request costs, but more memory pressure on the broker during reads. readBufferSize (default 1MB) tunes read throughput from the tier. Offload parallelism defaults to the broker's CPU core count.

For monitoring, the Prometheus metrics are the difference between a quiet cost saving and a silent backlog of un-offloaded data eating your BookKeeper disks. Watch pulsar_offloaded_storage_size (should grow steadily), pulsar_offload_errors_total (should be flat at zero), and pulsar_offload_latency_seconds. Alert on offload errors and on the gap between expected and actual offloaded size — a failing offloader doesn't stop writes, it just quietly stops saving you money until a disk fills.

The trade-offs worth knowing upfront

Geo-replication replicates the log, not the objects. If you run two clusters with tiered storage, each offloads independently to its own bucket. Cross-region durability of offloaded data requires bucket-level replication (e.g., S3 Cross-Region Replication), configured separately.

Ledger GC waits on offload. BookKeeper only garbage-collects a ledger after all its segments are offloaded and acknowledged. A very high offload threshold delays ledger deletion and keeps BookKeeper disks fuller than the retention policy suggests.

Schema is frozen at write time. Offloaded segments keep the schema version they were written with. Consumers replaying history across a schema evolution must handle multiple versions; nothing rewrites old data.

Encryption is two-layered. BookKeeper encryption and object-store encryption (SSE-S3/SSE-KMS) are independent. Rotating BookKeeper keys does not re-encrypt already-offloaded objects — plan key rotation for both layers.

A safe way to try it

Before touching production, run pulsar standalone locally with the filesystem offloader: set offloadersDirectory=./offload and a tiny threshold like 1MB, produce 10MB, and watch segment files appear in the offload directory while consumption from the tail still works. That ten-minute exercise validates your understanding of the read path without any cloud credentials. Configuration details and driver names vary slightly across Pulsar versions, so cross-check against the tiered storage docs for your specific release (the behavior described here reflects the 2.10–3.x line) before rolling out.

The decision rule is simple: if your retention window is measured in weeks and most reads are recent, tiered storage is close to free money. Just budget the monitoring to prove it keeps working.