Diagnosing Stalled Topic Compaction in Apache Pulsar

The Problem: Unbounded Storage Growth Despite Compaction

In Apache Pulsar, topic compaction is designed to retain only the latest message for any given key, preventing storage bloat in state-tracking topics. When compaction stalls, the ledger directory grows indefinitely, increasing disk pressure and potentially slowing down consumer read latency. The primary indicator is a discrepancy between the expected storage footprint (based on unique keys) and the actual disk usage on the BookKeeper nodes.

Quick Diagnostic Reference

Symptom	Likely Cause	Primary Metric/Check
Disk usage increasing; compaction_queue_depth is high	I/O Bottleneck or Lock Contention	BookKeeper Journal latency
compaction_queue_depth is 0; disk usage increasing	Misconfigured Retention Policy	Topic retention settings
Compaction runs but storage doesn't drop	High Key Cardinality / Frequent Updates	Unique key count vs. Total messages
Intermittent compaction failures	Disk Space Exhaustion (Ledger Dir)	df -h on Bookies

Step-by-Step Diagnostic Workflow

Follow these steps to identify why a topic is failing to compact efficiently. These checks assume you are running Pulsar 2.10+ and have access to the pulsar-admin CLI.

1. Verify Compaction Status

First, determine if the compaction process is actually queued or executing. Run this command from a broker or a machine with the Pulsar admin tools configured:

# Replace <tenant>/<namespace>/<topic> with your actual topic path
pulsar-admin topics compaction status persistent://tenant/namespace/topic

Expected Result: The output should indicate whether a compaction cycle is currently active. If the status is idle but the compaction_queue_depth metric in Prometheus is climbing, the broker is unable to schedule the merge operation.

2. Analyze Retention and Compaction Alignment

Compaction operates on a window. If your retention policy is too aggressive, Pulsar may delete messages before the compaction process can merge them, or conversely, a lack of a defined retention policy may prevent the trigger of a compaction cycle.

Check the topic's current policy:

pulsar-admin topics get-retention persistent://tenant/namespace/topic

Risk: If retentionTimeInMinutes is set shorter than the time it takes for a compaction cycle to complete, you may experience data loss for keys that haven't been updated recently.

3. Check BookKeeper Ledger Performance

Compaction is I/O intensive. It reads multiple ledgers and writes a new, compacted ledger. If your Bookies are using mmap ledgers, latency is generally lower, but if the journal disk is saturated, the merge operation will stall.

• Check for "Noisy Neighbor" effects: Monitor if other tenants are performing heavy writes to the same Bookie nodes.
• Check disk space: If the ledger directory is near capacity, the broker may fail to create the temporary ledgers required for the compaction merge.

Fixes Based on Findings

Finding: High Queue Depth / I/O Saturation

If the compaction queue is backed up and Bookie I/O is peaked, reduce the compaction frequency or move the topic to a dedicated namespace with isolated resource quotas to prevent cluster-wide latency spikes.

Finding: Misaligned Retention

Ensure the retention policy allows enough time for the compaction window to close. Adjust the retention to be slightly longer than your expected compaction cycle:

# Set retention to 24 hours (1440 minutes) and size to 10GB
pulsar-admin topics set-retention persistent://tenant/namespace/topic --size 10G --time 1440

Finding: Disk Space Exhaustion

If the ledger directory is full, you must increase disk capacity or manually trigger the deletion of old, unused ledgers via the BookKeeper admin tool. Note that manual ledger deletion is a high-risk operation and should only be done after verifying no consumers are lagging behind the deletion point.

Verification and Rollback

To verify the fix, monitor the ledger entry counts on the BookKeeper nodes. After a successful compaction cycle, the number of entries for a specific key across the ledgers should be reduced to one (the most recent version).

Rollback: If changing retention policies caused premature data loss, you cannot "undo" the deletion of compacted messages. To prevent further loss, immediately increase the retentionTimeInMinutes to a conservative value.