Enabling Pulsar Message TTL: A Decision Guide for Auto‑Cleanup vs. Compliance

Problem: Storage Pressure vs. Regulatory Retention

When a Pulsar cluster grows, disk usage can become a bottleneck. Traditional retention policies limit total storage or time per topic, but they don’t automatically prune older messages once the limit is hit. Pulsar’s Message TTL (Time‑To‑Live) feature deletes messages that have existed longer than a configured threshold, freeing space without manual intervention. However, TTL can conflict with compliance requirements that mandate keeping messages for a minimum period. This guide helps you decide whether to enable TTL, how to configure it safely, and how to validate the effect.

Decision: Enable TTL or Not?

Enable TTL when:

• Storage pressure is high and you have no legal obligation to retain all messages beyond a short window.
• Your application can tolerate occasional loss of unconsumed messages older than the TTL.
• You are running Pulsar 2.9 or newer on all brokers.

Do not enable TTL when:

• Regulatory or audit requirements mandate retention of every message for a specified period.
• Consumers rely on a guaranteed backlog of all messages, even if they are older.
• You operate on a cluster that lacks TTL support (pre‑2.9).

Constraints & Assumptions

• All brokers must be at least Pulsar 2.9.0 to support TTL on both normal and partitioned topics.
• TTL is configured per topic; it does not affect the broker‑wide retention policy.
• TTL cleanup occurs during compaction cycles, so messages may linger up to the compaction interval.
• Unacknowledged messages are retained until acknowledgment, regardless of TTL.
• Enabling TTL may increase broker CPU usage due to periodic checks.

Options Table

Option	Configuration	Effect on Storage	Impact on Compliance	Performance Impact
Retention Policy (time or size)	pulsar-admin topics set-retention my-topic --time 86400 --size 10GB	Deletes messages when total size or time limit is exceeded.	Can be tuned to meet compliance; may still keep older messages if size not reached.	Low CPU overhead; relies on broker compaction.
Message TTL	pulsar-admin topics set-property my-topic ttl 604800	Deletes all messages older than the TTL, regardless of total size.	May violate retention if TTL < required legal period.	Higher CPU due to periodic TTL checks; compaction still required.
Combined TTL + Retention	Set both ttl and retention properties.	Provides dual safety net: older messages removed by TTL, size/time limits enforced.	Compliance satisfied if TTL >= legal retention; otherwise, retention overrides.	CPU impact additive; monitor resource usage.

Trade‑Offs Explained

• Storage Savings vs. Data Loss – TTL guarantees removal of stale data, which is ideal for logs or metrics that only need recent values. If your use case requires a full audit trail, TTL is unsuitable.
• CPU Overhead vs. Disk I/O – Each TTL check scans message timestamps; frequent checks can spike CPU. Adjust the ttl-check-interval-ms broker config if needed.
• Backlog Integrity vs. Cleanup Speed – Unacknowledged messages are protected by TTL until consumers consume them. This protects consumer workloads but can delay cleanup.
• Compliance vs. Operational Simplicity – Retention policies are easier to audit because they’re explicit about size/time limits. TTL requires careful documentation of the TTL value to prove compliance.

Concrete Implementation & Validation

Step 1: Verify Pulsar Version

# Run on any broker node
pulsar-admin version
# Expect output like: Pulsar 2.10.0

Step 2: Create a Topic (if not existing)

# Normal topic
pulsar-admin topics create persistent://public/default/my-topic
# Or partitioned topic
pulsar-admin topics create-partitioned-topic persistent://public/default/part-topic 5

Step 3: Enable TTL

# Set TTL to 7 days (604800 seconds) for normal topic
pulsar-admin topics set-property persistent://public/default/my-topic ttl 604800
# For partitioned topic, apply to each partition
pulsar-admin topics set-property persistent://public/default/part-topic-0 ttl 604800
# Repeat for part-topic-1 … part-topic-4

Step 4: Produce Test Messages

# Produce 100 messages with a 1‑second interval
for i in {1..100}; do
  pulsar-client produce persistent://public/default/my-topic <<EOF
  {"msg": "$i"}
EOF
  sleep 1
done

Step 5: Wait for TTL Expiry

Wait at least TTL + compaction interval seconds. The default compaction interval is 3600 s, so wait 604800 + 3600 ≈ 608400 s (~168 h).

Step 6: Consume & Verify Deletion

# Consume all available messages
pulsar-client consume persistent://public/default/my-topic -s my-sub --max-consume 100
# Expect zero messages if TTL worked correctly

Alternatively, check broker stats:

# Before TTL expiry
pulsar-admin topics stats persistent://public/default/my-topic | grep "storedMessages"
# After TTL expiry
pulsar-admin topics stats persistent://public/default/my-topic | grep "storedMessages"
# The count should drop to 0

Step 7: Monitor Resource Impact

# On broker node, check CPU usage of pulsar broker process
top -p $(pgrep -f pulsar | head -1)
# Look for increased CPU during TTL cleanup periods

Limitations & Practical Checks

• TTL only affects persisted messages. In‑flight or unacknowledged messages remain until consumers acknowledge them.
• Deleted messages may still appear in the broker’s ledger for up to the compaction interval; they’re marked as deleted but not physically removed until a compaction cycle.
• If your cluster uses compaction-interval-ms > 24 h, cleanup can be delayed; consider reducing this setting if frequent cleanup is needed.
• Always audit the pulsar-admin topics stats output before and after TTL to confirm behavior.
• For compliance‑heavy workloads, maintain a separate audit log or use Pulsar’s Message Key feature to tag retention‑critical messages and avoid TTL on those topics.

Conclusion

Message TTL is a powerful tool for auto‑cleaning stale data and mitigating storage pressure in Pulsar. Use it when your use case tolerates the loss of unconsumed old messages and when your compliance framework allows a short retention window. Pair TTL with a traditional retention policy for extra safety, and always monitor broker CPU and compaction activity to ensure the system behaves as expected.