Apache Pulsar Tiered Storage: Architecture Decisions and Operational Reality

The Problem: Hot Storage Cost vs. Cold Data Access

BookKeeper provides the durability and low-latency reads Pulsar needs for active topics, but keeping months of data on SSD-backed BookKeeper ensembles gets expensive fast. Tiered storage moves sealed ledger segments to object storage (S3, GCS, Azure Blob) after a configurable threshold, keeping hot data on BookKeeper and cold data on cheaper storage—without changing client code or breaking ordering guarantees.

Takeaway: Enable tiered storage with a single offloader, a size- or time-based threshold, and broker IAM permissions to the bucket. Monitor offload lag and BookKeeper disk usage; expect reads to transparently fetch from object storage when segments aren't local. The design changes only if you need sub-millisecond tail latency for all data, face regulatory residency constraints, or have write-once/read-never workloads better served by Pulsar IO sinks.

Requirements Driving the Design

• Cost separation: BookKeeper on NVMe for recent data; object storage for the long tail.
• Read-path transparency: Existing producers/consumers require zero changes. Ordering and exactly-once semantics hold.
• Durability authority stays with BookKeeper: The write quorum (ensemble) remains the source of truth. Object storage is a read replica, not a write target.
• No BookKeeper/ZooKeeper schema changes: Offloading is a broker-side operation coordinated via ZooKeeper metadata.

Smallest Viable Configuration

On a 3-broker cluster (Pulsar ≥ 2.10.4 or 2.11.1+), add to broker.conf (or standalone.conf for dev):

# Enable tiered storage
tieredStorageEnabled=true

# Offloader implementation: s3, gcs, or azure-blob
offloadersDirectory=./offloaders
tieredStorageDriver=org.apache.pulsar.tieredstorage.offloaders.S3Offloader

# Threshold: offload when segment exceeds 1 GB OR is older than 7 days
offloadThreshold=1G
offloadThresholdTime=7d

# Bucket and credentials (IAM role preferred over keys)
s3ManagedLedgerOffloadBucket=my-pulsar-offload
s3ManagedLedgerOffloadRegion=us-east-1
# s3ManagedLedgerOffloadAccessKey=...  # omit if using instance profile
# s3ManagedLedgerOffloadSecretKey=...

# Throttle to protect serve path
offloadMaxConcurrentTasks=2
offloadThrottleRate=100MB/s

Where to apply: Every broker node. Restart brokers rolling (one at a time) to pick up changes. Requires broker process write access to the bucket (PutObject, GetObject, ListBucket).

Verification command (run from any broker host with pulsar-admin in PATH):

pulsar-admin topics offload-status persistent://public/default/my-topic
# Expected: shows OFFLOADED segments with object-store URIs once threshold met

Trust and Data Boundaries

Layer	Authority	Notes
Write path	BookKeeper ensemble (write quorum)	Acknowledgment returned only after quorum ack. Offloader never participates in write ack.
Compaction/deduplication	BookKeeper-resident data only	Runs before offload. Offloaded segments are immutable; duplicates may remain in cold tier.
Read path	Broker redirects or proxies	If segment is local → BookKeeper read. If offloaded → broker fetches from object storage (signed URL or server-side) and streams to client. Ordering preserved.
Geo-replication	Per-cluster offload	Replication copies BookKeeper data. Each remote cluster offloads independently to its own bucket.

The offloader runs with broker privileges but does not acknowledge writes. Corrupted object manifests are detected on read via checksum; broker falls back to BookKeeper if the segment is still present.

Operational Checks and Metrics

Key Prometheus Metrics

• pulsar_tiered_storage_offloaded_bytes_total — cumulative bytes moved to object storage.
• pulsar_tiered_storage_offload_latency_seconds — histogram of offload task duration.
• pulsar_tiered_storage_offload_failed_total — failed offload tasks (alert on > 0).
• pulsar_managed_ledger_offloaded_segments — segments currently offloaded per topic.

Alerting Rules (example)

# Offload lag: data older than 2x threshold still on BookKeeper
- alert: PulsarOffloadLag
  expr: pulsar_managed_ledger_oldest_segment_age_seconds > (7 * 24 * 3600 * 2)
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Offload lag exceeds 2x threshold"

# Failed offload tasks
- alert: PulsarOffloadFailures
  expr: rate(pulsar_tiered_storage_offload_failed_total[5m]) > 0
  for: 5m
  labels:
    severity: critical

Manual Verification

1. Produce ~50 MB to a topic with retentionTime=1h, offloadThreshold=10MB.
2. Wait for offload (watch offload-status or bucket).
3. Consume from head (oldest) and tail (newest) — both should succeed transparently.
4. Check bucket: segment files + manifest.json per ledger.

Failure Modes

Scenario	Impact	Recovery
Object storage outage	New offloads block; reads/writes to BookKeeper continue	Offloader retries idempotently when bucket recovers. No data loss.
Network partition during offload	Partial segment upload	Manifest written only on complete upload. Incomplete segments stay in BookKeeper; retry picks up.
Corrupted object / manifest	Read fails for that segment	Broker detects checksum mismatch, falls back to BookKeeper if segment not yet deleted by retention.
Broker restart mid-offload	In-flight task interrupted	Offload state persisted in ZooKeeper; resumes on broker restart. Verify with offload-status.
Bucket TTL < Pulsar retention	Objects deleted before Pulsar expires them	Set bucket lifecycle TTL > Pulsar retentionTime. Deleting topic removes both tiers.

Conditions That Would Change the Design

• Sub-millisecond tail latency for all data: Keep everything on BookKeeper/SSD. Tiered storage adds object-storage RTT (typically 5–50 ms) for cold reads.
• Regulatory data residency: If object storage region violates sovereignty, disable tiered storage or use on-prem S3-compatible storage (MinIO, Ceph RGW) in the same region.
• Write-once/read-never (compliance archive): Consider Pulsar IO sinks writing directly to object storage/Parquet instead of tiered storage—avoids BookKeeper write path entirely.
• Heavy compaction workloads: Compacted topics retain duplicates in cold tier. If deduplication is critical, keep compacted topics on BookKeeper longer (raise offloadThresholdTime).
• Broker memory pressure during large offloads: Lower offloadMaxConcurrentTasks (default 2) and offloadThrottleRate. Monitor broker JVM heap; offload tasks allocate buffers proportional to segment size.

Limitations and Version Constraints

• Pulsar < 2.10 had bugs in offload cancellation and manifest consistency. Production requires ≥ 2.10.4 or ≥ 2.11.1.
• Schema evolution and compaction operate only on BookKeeper-resident data. Offloaded segments are immutable.
• Geo-replication does not replicate offloaded segments; each cluster offloads independently.
• Retention policies apply to the combined tier. Deleting a topic removes both BookKeeper and object storage data.

Practical Verification Checklist

1. Deploy 3-broker cluster with BookKeeper on local disks and MinIO (S3-compatible) for test.
2. Create topic: pulsar-admin topics create persistent://public/default/test-tiered -r 1h -o 10MB.
3. Produce 50 MB: pulsar-perf produce persistent://public/default/test-tiered -n 50000 -s 1000 -r 10000.
4. Wait for offload, then verify: pulsar-admin topics offload-status persistent://public/default/test-tiered shows OFFLOADED.
5. Consume head and tail: pulsar-perf consume ... -ss Earliest and -ss Latest.
6. Check MinIO bucket for segment files + manifest.json.
7. Simulate broker restart during offload; confirm resume and no data loss.
8. Monitor pulsar_tiered_storage_offloaded_bytes_total and pulsar_tiered_storage_offload_failed_total for 24h.

If all steps pass, the tiered storage path is operational. Adjust thresholds and throttling based on observed BookKeeper disk usage and offload latency percentiles.