Configuring Tiered Storage in Apache Pulsar Namespaces

Desired Outcome

Offload older Pulsar messages to an external object store (e.g., Amazon S3) so that recent data stays on the local BookKeeper cluster and storage costs are reduced.

Prerequisites

• Apache Pulsar cluster running version 2.8 or newer (tiered‑storage offloading is available since 2.8).
• An offloader configuration in broker.conf (for example the AWS S3 offloader with managedLedgerOffloadDriver=aws-s3 and bucket credentials).
• Network connectivity from broker nodes to the object-store endpoint.
• Administrative access to run pulsar-admin commands.

Procedure

1. Verify Pulsar version

   bin/pulsar version

   Confirm the output includes version 2.8.x or newer.

2. Set the offload threshold for a namespace

   pulsar-admin namespaces set-offload-threshold tenant1/my-namespace 10GB

   When a ledger exceeds 10 GB the offloader will start moving segments to the configured object store.

3. Set the offload deletion lag

   pulsar-admin namespaces set-offload-deletion-lag tenant1/my-namespace 2d

   Segments will remain in the object store for at least two days before they are eligible for deletion.

4. Verify the offload threshold

   pulsar-admin namespaces get-offload-threshold tenant1/my-namespace

   Expected output includes the size limit you just set.

5. Verify the deletion lag

   pulsar-admin namespaces get-offload-deletion-lag tenant1/my-namespace

   Output should show the time window you configured.

6. (Optional) Apply a retention policy

   pulsar-admin namespaces set-retention tenant1/my-namespace --time 30d --size 10GB

   Keeps the last 30 days or 10 GB locally; older data is eligible for the offload rules above.

7. Produce and consume messages to confirm normal operation

   Send a few messages with pulsar-client and read them back. After the threshold is exceeded, check the object store for uploaded segments.

Expected Checks

• Broker logs show messages from the ManagedLedgerOffloader moving segments and uploading to the bucket.
• Object store contains keys that follow the Pulsar namespace‑based naming convention (e.g., tenant1/my-namespace/partition0/ledger-...).
• pulsar-admin namespaces get-offload-threshold tenant1/my-namespace returns the size limit you set.
• pulsar-admin namespaces get-offload-deletion-lag tenant1/my-namespace returns the time window.
• Consumers can still read older messages; if a segment has been deleted from both tiers a NOT_FOUND error is returned.

Recovery / Rollback Options

• To disable offloading, reset the threshold to zero:

  pulsar-admin namespaces set-offload-threshold tenant1/my-namespace 0

  This stops new segment uploads to the object store.
• To change the deletion lag, run the set-offload-deletion-lag command with a new value or 0 to allow immediate deletion.
• If the object-store credentials change, update the broker.conf entries (managedLedgerOffloadDriver, s3ManagedLedgerOffloadBucket, etc.) and restart the affected brokers.
• To revert retention limits, adjust the set-retention command accordingly.

Limitations & Practical Considerations

• Offloading adds I/O overhead; monitor broker latency, especially during large segment uploads.
• Network bandwidth to the object store can become a bottleneck; consider throttling or using a dedicated link.
• Object-store costs (storage, request, data-egress) should be evaluated against the savings from reduced BookKeeper usage.
• Schema evolution is independent of offloading, but consumers reading archived data must support the schema version that was active when the segment was offloaded.
• Disabling offload does not automatically delete objects already in the bucket; manual cleanup is required if you want to remove them.

Verification Checklist

Check	How to Verify
Pulsar version	Run bin/pulsar version; look for 2.8+
Offload threshold set	pulsar-admin namespaces get-offload-threshold tenant1/my-namespace
Offload deletion lag set	pulsar-admin namespaces get-offload-deletion-lag tenant1/my-namespace
Objects in bucket	List bucket keys via AWS CLI (aws s3 ls) or the object-store console
Consumer read from tiered segment	Produce beyond the threshold, then consume with pulsar-client --topic tenant1/my-namespace; observe any latency or NOT_FOUND if the segment was deleted.