Decision Guide: Enabling Apache Pulsar Tiered Storage for Cost-Effective Message Archiving

Decision and Constraints

You need to reduce storage costs for a Pulsar cluster while keeping recent messages available for low-latency consumption. The decision is whether to enable Tiered Storage for a given topic or namespace. Constraints include:

• Required Pulsar version 2.8 or later (tiering was introduced in 2.8).
• Acceptable read latency for older messages (object-storage latency vs. broker SSD/RAM).
• Retention policy: you may still want TTL-based deletion after a certain period.
• Operational overhead: managing credentials, bucket lifecycle, and monitoring tiering transfers.

Comparison of Options

Option	Description	When to Choose
No tiering (all data on broker)	Messages remain on BookKeeper ledgers (SSD/RAM) until TTL expires or manual purge.	Latency-critical workloads where any increase in read latency is unacceptable, or you are on a Pulsar version earlier than 2.8.
Enable tiered storage	Configurable age threshold moves older segments to an external object store (S3, GCS, Azure Blob, etc.). Recent data stays on fast tiers.	Cost-saving is a priority, you can tolerate higher latency for historic reads, and you run Pulsar 2.8+ with proper object-storage credentials.

Trade-offs

Latency vs. Cost

Tiered storage shifts the storage medium from low-latency BookKeeper to higher-latency object storage. Reads of tiered-offloaded messages incur the object store's GET latency (typically tens to hundreds of milliseconds) plus any network hop. Recent messages (below the threshold) retain sub-millisecond to low-millisecond latency.

Operational Complexity

Enabling tiering adds:

• Credential management (access keys, IAM roles, or service accounts) for the object store.
• Monitoring of tiering transfer logs to detect throttling or failures.
• Potential need to adjust bucket lifecycle rules if you also rely on TTL for deletion.

Metadata Overhead

Each offloaded segment creates metadata entries in the broker. Setting the threshold too low (e.g., offloading after a few minutes) can increase metadata operations and affect broker throughput. A common starting point is offloading after several hours or a day, depending on message rate.

Concrete Implementation

The following steps show how to enable tiered storage for a namespace tenantA/app1 using AWS S3 as the object store. Adjust placeholders for your environment.

Prerequisites

• Pulsar 2.8 or later installed and accessible via pulsar-admin.
• An S3 bucket (e.g., pulsar-tiered-archive) with a policy granting the Pulsar broker read/write access.
• Broker configuration managedLedgerDefaultMarkDeleteRate and managedLedgerMaxSizePerLedger tuned for your workload (keep defaults unless you have specific needs).

Step 1: Create a Storage Class Definition

Define a YAML snippet that tells Pulsar how to reach S3. Save it as s3-tiered-storage.yaml:

# s3-tiered-storage.yaml
storageClassName: s3-offload
s3:
  endpoint: s3.amazonaws.com
  bucket: pulsar-tiered-archive
  region: us-east-1
  # Use IAM role attached to the broker or provide accessKey/secretKey
  # accessKey: YOUR_ACCESS_KEY
  # secretKey: YOUR_SECRET_KEY
  # Optional: enable server-side encryption
  # sse: true

Step 2: Apply the Storage Class to the Namespace

Run the following command (requires broker admin privileges):

pulsar-admin namespaces set-offload-policies \
  --storage-class s3-offload \
  --threshold 1h \
  tenantA/app1

Explanation:

• --storage-class references the class defined in the YAML (the broker must have been started with --offloadDriversDir pointing to a directory containing this file).
• --threshold 1h means messages older than one hour are eligible for offloading.
• The command updates the namespace's offload policy; existing topics inherit the setting immediately.

Step 3: Verify Configuration

Check that the policy is active:

pulsar-admin namespaces get-offload-policies tenantA/app1

Look for output showing the storage class and threshold. No actual data movement occurs until segments age past the threshold.

Step 4: Monitor Tiering Activity

Observe broker logs for lines like:

INFO  OffloadDriver - Offloading ledgerId=12345 to s3://pulsar-tiered-archive/...

You can also query the status of a specific topic:

pulsar-admin topics tiering-status persistent://tenantA/app1/my-topic

This returns the percentage of ledger storage offloaded and any error codes.

Validation and Limitations

To confirm that tiering is working as expected:

1. Produce a steady stream of messages to a test topic in the namespace.
2. Wait longer than the threshold (e.g., more than 1 hour) and then attempt to consume a message from the early part of the stream.
3. Measure the end-to-end latency; it should be higher than for recent messages but still within your acceptable range.
4. Check the S3 bucket for objects matching the expected naming pattern (pulsar-offload-...).

Limitations to consider:

• If the threshold is set too low, the broker may generate excessive offload requests, increasing CPU and network usage.
• Object-storage latency can vary; transient network spikes may cause occasional read timeouts for historic data.
• Mis-matched IAM permissions lead to silent publish failures; always test credentials with a minimal offload before applying to production topics.
• TTL deletion and tiering can coexist, but ensure TTL is longer than the offload threshold if you want to retain archived data for a period before deletion.

Rollback Considerations

Disabling tiering does not automatically move data back from object storage to BookKeeper. To revert:

1. Update the namespace offload policy to a threshold of 0 (or remove the policy) using pulsar-admin namespaces set-offload-policies --threshold 0.
2. Existing offloaded segments remain in S3; new segments will stay on fast tiers.
3. If you need to retrieve historic data, you must consume directly from S3 via Pulsar's read-offload API or copy the objects back manually.

Because the operation only changes where new segments are stored, a formal rollback of already-offloaded data is not required unless you want to repatriate it, which is a separate migration task.