Choosing Between Pulsar Message TTL and Topic Retention for Data Lifecycle Management

Problem and Takeaway

Apache Pulsar offers two distinct mechanisms for controlling how long messages stay in a topic: message TTL (time‑to‑live) and topic retention (namespace‑level storage policy). Both delete data, but they operate at different layers and serve different compliance or operational goals. The quick decision: use message TTL when you need automatic, per‑message expiration that frees space as soon as the TTL elapses; use topic retention when you must guarantee a minimum retention window for audit, replay, or regulatory reasons, even if consumers have already acknowledged the messages.

Decision Constraints

• Deletion granularity – TTL applies to each message individually; retention applies to the whole topic/namespace.
• Consumer acknowledgment – TTL ignores ack state; retention also ignores ack state but enforces a minimum time or size window.
• Storage predictability – Retention gives a predictable upper bound (time + size); TTL can free space earlier but may lag under load.
• Operational scope – TTL is set per‑topic (or per‑namespace default); retention is configured at namespace level and inherited by all topics unless overridden.
• Compliance requirements – Retention is the natural fit for “keep for N days” mandates; TTL is better for “expire stale events quickly”.

Supported Options at a Glance

Feature	Message TTL	Topic Retention
Configuration level	Topic (or namespace default)	Namespace (inherited by topics)
Deletion trigger	Message age exceeds TTL	Time window expires or size limit reached
Effect on acknowledged messages	Deleted regardless of ack	Deleted regardless of ack
Space reclamation speed	Fast (seconds‑to‑minutes, broker‑dependent)	Bounded by configured window; may hold data longer
Typical use case	Event‑stream cleanup, cache‑like topics	Audit logs, regulatory replay, disaster recovery
Risk of premature loss	High if TTL < processing latency	Low (window protects against early loss)

Trade‑offs

Message TTL

TTL is enforced by the broker when a message’s publish timestamp plus the TTL passes. The broker does not wait for consumer acknowledgment, so space is reclaimed quickly. However, deletion is not instantaneous; under heavy load the broker may lag by seconds to minutes. If the TTL is shorter than the end‑to‑end processing time, consumers can miss messages entirely – a classic data‑loss scenario.

Topic Retention

Retention guarantees that messages survive for at least the configured time (e.g., 7 days) or until the topic reaches a size ceiling (e.g., 500 GB). This predictability is valuable for compliance, but it can cause temporary storage spikes because messages linger even after all consumers have processed them. Retention changes affect every topic in the namespace unless you override a specific topic, so a mis‑configuration can inflate storage across many streams.

Combining Both

A common pattern is to set a short TTL (e.g., 1 hour) to prune stale events quickly, while configuring a longer retention window (e.g., 30 days) to satisfy audit requirements. The TTL removes messages from the active ledger early; the retention policy keeps the ledger segments on disk until the window expires, after which compaction or ledger rollover finally frees the space.

Concrete Implementation & Validation

1. Set Message TTL on a Topic

# Run on a machine with pulsar-admin in PATH and admin permissions
pulsar-admin topics set-message-ttl \
  --ttl 3600 \
  persistent://my-tenant/my-ns/my-topic

Replace 3600 with the desired TTL in seconds. The command updates the topic’s metadata; no broker restart is required.

2. Configure Namespace Retention

pulsar-admin namespaces set-retention \
  --time 7d \
  --size 500G \
  my-tenant/my-ns

This applies a 7‑day time window and a 500 GB size limit to every topic under my-tenant/my-ns. To override a single topic, use pulsar-admin topics set-retention with the same flags.

3. Verify TTL Behavior

1. Publish a test message:

   pulsar-client produce persistent://my-tenant/my-ns/my-topic \
     --messages "ttl-test-$(date +%s)" \
     --num 1

2. Wait slightly longer than the TTL (e.g., 1 hour + 5 minutes).
3. Check topic stats:

   pulsar-admin topics stats persistent://my-tenant/my-ns/my-topic

4. Confirm msgRateIn is zero and storageSize has dropped, indicating the message was removed.

Risk: If the broker is under heavy load, the deletion may not appear immediately; allow a few extra minutes before concluding failure.

4. Verify Retention Behavior

1. Publish enough messages to approach the size limit (or simply wait for the time window).
2. After the configured retention period (e.g., 7 days), run the same topics stats command.
3. Observe that storageSize reflects only messages younger than the window; older segments should be gone or marked for compaction.

Risk: Retention does not trigger immediate deletion at the exact moment the window expires; the broker’s retention manager runs periodically, so a small lag is normal.

5. End‑to‑End Consumer Check (Optional)

# Java client snippet (pseudo‑code)
Consumer consumer = client.newConsumer()
    .topic("persistent://my-tenant/my-ns/my-topic")
    .subscriptionName("ttl-verification")
    .subscribe();

Message msg = consumer.receive(5, TimeUnit.SECONDS);
if (msg == null) {
    System.out.println("No message delivered – likely expired by TTL");
} else {
    System.out.println("Received: " + new String(msg.getData()));
}
consumer.close();

Run this after the TTL interval; a null return confirms the message was not delivered because it was purged.

Limitations & Practical Checks

• TTL deletion lag: monitor msgDropRate in topic stats to detect backlog.
• Retention size limit is a soft ceiling; the broker may temporarily exceed it during ledger rollover.
• Changing retention on a live namespace affects all topics instantly – test in a staging namespace first.
• Compaction and ledger rollover are separate processes; they ultimately reclaim the physical disk space after logical deletion.

To validate the combined setup, create a test namespace, apply a 1‑hour TTL and a 30‑day retention, publish a burst of messages, then verify that messages disappear from consumer view after 1 hour while the topic’s storage size remains stable until the 30‑day window passes.

Diagram Labels

• Message TTL → per‑message expiration
• Topic Retention → namespace time/size window
• Combined policy → fast prune + audit window
• Broker retention manager → periodic cleanup