The short answer: in Apache Pulsar, broker stability during cloud offload problems comes from treating offload as a background, retryable task and keeping BookKeeper as the source of truth. The offload driver never blocks foreground writes, and ledger metadata in ZooKeeper (or the configured metadata store) is only updated after the object store confirms the data, so a slow or failing S3/GCS backend degrades offload progress — not correctness.
Likely explanation vs. confirmed behavior
Confirmed behavior (well-established Pulsar design):
- Offload runs asynchronously per ledger. A ledger is only marked offloaded after the driver completes the upload and the managed-ledger metadata is updated. Until then, reads are served from BookKeeper exactly as before.
- If the offload fails (throttling, 403s from an expired IAM session, network timeouts), the ledger simply remains in BookKeeper. Retention of the local copy is governed separately, so data is not lost because an upload failed.
- Reads of already-offloaded data go through the broker, which fetches from the object store. High provider latency shows up as slow reads for consumers reading cold data — it does not stall producers or hot consumers.
Likely explanation for instability reports: most production issues come from misaligned thresholds (offloading too aggressively so hot reads hit the remote tier) or undersized read buffers/connection pools against the object store, not from the offload mechanism itself.
Configuration for this case
In broker.conf (or the operator CRD equivalent):
managedLedgerOffloadDriver=aws-s3 # or google-cloud-storage
s3ManagedLedgerOffloadBucket=my-bucket
s3ManagedLedgerOffloadRegion=us-east-1
s3ManagedLedgerOffloadRole=my-offload-role # IAM, not static keys
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864
s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576
Then control when offload triggers, per namespace or topic:
pulsar-admin namespaces set-offload-threshold my-tenant/my-ns --size 10G
pulsar-admin namespaces set-offload-deletion-lag my-tenant/my-ns --lag 24h
Practical recommendations for high-latency or flaky providers:
- Set the offload threshold with headroom. Trigger offload well before BookKeeper disks fill, so retries have time to succeed. A threshold that fires at 90% disk usage leaves no slack for a multi-hour provider outage.
- Keep a deletion lag. The deletion lag delays removal of the local BookKeeper copy after offload. It is your safety net against both provider inconsistency and offload bugs — size it to your recovery objective, not to zero.
- Use IAM roles, not static credentials. Temporary access denials are frequently expired or rotated keys; instance/workload identity removes that failure class entirely.
- Tune read buffers for cold reads. Consumers reading offloaded data are latency-bound on the object store; larger read buffers and prefetch reduce round trips.
Metadata consistency during transition
Pulsar does not rely on the object store's consistency model for correctness. The sequence is: upload all blocks of the ledger, then atomically update the managed-ledger metadata to point at the offloaded copy. Because the metadata store is strongly consistent, a broker either sees the ledger as local (reads from BookKeeper) or offloaded (reads from the object store) — never a half-uploaded state. Eventual consistency in the object store is absorbed by the fact that metadata flips only after the driver has confirmed the upload.
Verification
- Check offload status per topic:
pulsar-admin topics stats-internal my-tenant/my-ns/persistent/my-topic and inspect ledgers[].offloaded. - Confirm objects exist in the bucket under the expected prefix before lowering the deletion lag.
- Simulate a provider outage (block egress to the endpoint briefly) and confirm producers keep writing and ledgers offload after recovery.
Note: exact property names vary across Pulsar versions and offload drivers; verify against the documentation for your deployed version. One detail worth confirming for your environment: which Pulsar version and whether you run brokers under an operator, since that changes where these keys are set.
pulsar