Time-based or size-based retention for replay data before a Pulsar rollback?

The Recommended Policy

To guarantee a replay horizon under bursty traffic without exceeding bookie SSD budgets, use a hybrid retention policy: set a time-based retention window that matches your maximum rollback horizon, and a size-based cap that is calculated to accommodate peak traffic bursts for that same duration.

How Retention and Offload Interact

In Pulsar 3.x, the broker enforces the stricter of the two retention limits. If you rely solely on size-based retention, a traffic spike will evict older ledgers prematurely, breaking your replay horizon. If you rely solely on time-based retention, a spike can exhaust bookie SSDs.

Tiered storage offload solves this by moving data to cheaper storage before the retention policy deletes it. To keep data local (on SSDs) for the immediate rollback window, configure the offloadThreshold to be slightly larger than your expected rollback lag. This ensures ledgers remain in BookKeeper until the window closes, while the overall retention policy prevents the total dataset from growing indefinitely.

Implementation Steps

1. Calculate Size Cap: Determine the maximum possible data volume produced during a burst over your desired rollback window (e.g., Peak Throughput × Window Duration).
2. Set Namespace Retention:

   pulsar-admin namespaces set-retention / --size  --time 

3. Configure Offload Threshold: Set the offload threshold to ensure data stays local for the duration of the rollback window before moving to tiered storage.

   pulsar-admin namespaces set-offload-threshold / --size 

4. Verify Local Residency: Use the following command to ensure ledgers are not offloaded prematurely:

   pulsar-admin topics list  --include-offloaded

Broker Failover and Offload Behavior

Offloader state is persisted in ZooKeeper. During a broker failover, the new broker owner resumes the offloading process based on the existing metadata. It will not prematurely offload ledgers that have not yet met the offloadThreshold. However, offloading may be delayed until the offloader component fully initializes on the new broker.

Verification and Assumptions

This guidance assumes Pulsar 3.x. Because offload behavior was refined in version 3.1.0, users on earlier 3.x releases should verify that offloadThreshold is being honored by monitoring ledger movement during a simulated burst.

Missing Diagnostic: What is the average ledger size (maxLedgerSize) for your topics? If ledgers are exceptionally large, they may be offloaded based on size thresholds before the time-based rollback window has elapsed.