Diagnosing Premature Tiered Storage Offload in Apache Pulsar

Symptom: recent reads are slower than expected

If consumers or readers see elevated latency for messages published within the last few minutes or hours, those messages may already reside in object storage instead of BookKeeper. Apache Pulsar tiered storage automatically moves older ledger segments from BookKeeper—the distributed log layer that stores Pulsar message data—to object storage such as S3 or GCS. The move is controlled by per-topic thresholds. When the thresholds are too low, recently published data is treated as cold data, and every read must traverse the object-storage API.

The useful takeaway is to check the topic’s tiered-storage policy before tuning consumers or scaling brokers. Raising the offload thresholds usually stops future premature offloads, but segments already moved to object storage generally stay there.

Cause and diagnostic signals

Observed condition	Likely cause	Where to look
Messages younger than expected are in object storage	offloadTimeMs or offloadSizeMB is set too low	Topic tiered-storage policy
All topics in a namespace offload early	Namespace-level policy overriding topic settings	Namespace tiered-storage policy
Offload errors in broker logs	Object-storage credentials, bucket policy, or TLS misconfiguration	Broker logs and bucket IAM
Offload happens immediately after segment creation	Segment size below offloadSizeMB with aggressive time threshold	Broker metrics offload.requests

Check 1: Read the topic tiered-storage policy

Run this from a machine with the Pulsar admin CLI and access to the broker. You need superuser or namespace-admin permissions. This is a read-only command.

pulsar-admin topics get tiered-storage --tenant <tenant> --namespace <namespace> --topic <topic>

Replace <tenant>, <namespace>, and <topic> with your values.

Expected check: the response should show offloadTimeMs and offloadSizeMB. If offloadTimeMs is smaller than your hot-data retention window—for example, 86400000 ms (one day) when you need two days of hot data—the policy is too aggressive. If the topic shows no explicit policy, it may inherit from the namespace.

Check 2: Confirm whether offload already occurred

Compare message publish timestamps against offloadTimeMs. If messages younger than the threshold appear in object storage, either the threshold changed recently or a namespace policy is overriding the topic setting. A reader fetching an offloaded segment must wait for the object-storage GET to complete, which adds network round-trips and can spike p99 latency.

You can also monitor broker metrics such as offload.requests and object.storage.bytes.written through JMX or your metrics dashboard. A spike shortly after publishing indicates premature offload.

Check 3: Check namespace inheritance

If the topic policy looks correct but offload still happens early, inspect the namespace-level tiered-storage configuration. Namespace defaults can apply to all topics unless overridden at the topic level.

Fix: Raise offload thresholds for hot topics

Update the topic policy so recent messages remain on BookKeeper longer. Choose values based on your hot-data window. For example, if you need messages from the last 48 hours available locally, set offloadTimeMs to at least 172800000 ms.

pulsar-admin topics set tiered-storage --tenant <tenant> --namespace <namespace> --topic <topic> --offload-time-ms <new-offload-time-ms> --offload-size-mb <new-offload-size-mb>

This command changes cluster state. Required permissions: superuser or namespace admin. Relevant risks:

• Already-offloaded segments remain in object storage; this command prevents future offloads but does not recall data.
• Setting offloadSizeMB too high can delay offload indefinitely and increase BookKeeper storage costs.
• Setting offloadTimeMs too high defers cost savings from tiered storage.

Rollback: re-run the same command with the previous offload-time-ms and offload-size-mb values. Record the original values before changing them.

Fix: Correct object-storage access if offload fails

If broker logs show authentication or access-denied errors, update the bucket policy so the Pulsar service account can read and write objects, and confirm TLS is enabled for the object-storage endpoint. This does not change Pulsar topic configuration, but failed offloads can leave segments stuck and generate repeated retry traffic.

Escalation criteria

Escalate to the platform or storage team when:

• Offload continues after threshold changes, suggesting namespace-level inheritance or a broker issue.
• Broker logs show repeated object-storage authentication or network errors.
• Read latency remains high after thresholds are raised, which may mean readers are still fetching cold objects or the working set is larger than BookKeeper capacity.
• You need to move already-offloaded segments back to BookKeeper; Pulsar does not generally provide a one-click recall operation.

Limitations and practical verification

Tiered storage became generally available in Pulsar 2.8. Pulsar 3.x adds incremental offload and compression support; behavior and metric names may differ by version. Confirm your broker version before applying this guide, and review release notes for your exact version because compression and incremental offload can change which segments become offload candidates.

Object storage reads have higher latency than BookKeeper reads by design. Tiered storage reduces long-term storage cost; it does not make cold reads faster.

Practical verification: after changing thresholds, publish a test message and confirm that reads remain fast for a period longer than the previous threshold but shorter than the new one. Then check offload.requests to confirm no new offload occurred for that segment during the hot window.