Apache Pulsar and S3 Tiered Storage: Balancing Offload Thresholds with Object Store Costs

Recommended strategy

For Pulsar 2.10+ with BookKeeper 4.7+, choose offloadThreshold so that low-throughput topics create only as many S3 objects as your request budget allows. Pulsar offloads a closed ledger segment to S3 once it exceeds the threshold, and each segment becomes one object. The broker does not pack small segments together, so a very low threshold multiplies PUT requests without reducing the total bytes stored.

Confirmed behavior

• offloadThreshold is a per-ledger size limit, not a global disk target.
• Pulsar writes one S3 object per eligible ledger segment; it does not auto-merge or auto-compress segments before offload.
• BookKeeper segmentSize controls when a ledger rolls; offload happens after the segment is closed and exceeds the threshold.

Why the trade-off exists

S3 charges per PUT/GET request and per GB stored. BookKeeper disk is usually billed as a flat monthly rate per GB. Aggressive offloading trades predictable disk expense for variable API expense. At minimal throughput, segment count can dominate the bill even when total payload is small.

Steps for this case

1. Measure average daily message bytes for the affected topics from Pulsar broker metrics.
2. Estimate daily segments using dailyBytes / segmentSize. Verify your BookKeeper segmentSize; defaults vary by release.
3. Multiply estimated daily segments by your AWS region's S3 PUT price.
4. Compare that request cost against the monthly BookKeeper disk cost for the same data. If requests dominate, raise offloadThreshold or increase segmentSize.
5. Start with offloadThreshold between 128 MB and 256 MB for low-traffic topics. Move toward 512 MB or higher if estimated daily segments reach the thousands.
6. Enable enableAutoOffload only when disk pressure is the main constraint, and tune autoOffloadThreshold as a disk-usage percentage.
7. Monitor Pulsar offload metrics and S3 request counts after any change.

Missing diagnostic detail

What is the average daily message volume in bytes for these topics? That figure determines whether 128 MB, 256 MB, or 512 MB is the correct minimum threshold.

Assumptions and uncertainty

This guidance assumes Pulsar 2.10+ and BookKeeper 4.7+. Parameter names and defaults can differ by release, so verify against your running version. S3 pricing varies by region and storage tier; cross-region data transfer costs are not included. Aggressive offloading can increase consumer read latency because S3 retrieval is slower than BookKeeper reads.