Log compaction with cleanup.policy compact,delete and keyless message handling

Keyless record handling under cleanup.policy=compact,delete

\n

When a topic uses cleanup.policy=compact,delete, the broker’s log cleaner follows a two‑phase approach. Compaction only processes records that carry a non‑null key; records with a null key are ignored by the compaction engine. Consequently, keyless messages are never rewritten or merged. They survive only as long as the segment that contains them remains below the delete‑retention thresholds (retention.ms or retention.bytes).

\n\n

Compaction precedence vs. delete retention

\n

\n
• Compaction runs on segments that have exceeded segment.ms and where at least min.cleanable.dirty.ratio (default 0.5) of the records are “dirty” (i.e., have a newer version for the same key). The cleaner rewrites the segment, keeping the latest value per key.
\n
• Delete retention evaluates every segment after it ages past retention.ms or exceeds retention.bytes and simply removes the entire segment, regardless of key presence.
\n

\n

Because keyless records are excluded from compaction, the delete‑retention path is the only mechanism that can purge them. Compaction may run on the same segment earlier if the dirty‑ratio condition is met, but it does not affect the eventual deletion of keyless messages; the segment will be deleted once it becomes old enough for retention.

\n\n

Bounding compaction timing and I/O in a test environment

\n

In an integration test without production credentials, you can influence both timing and disk impact by adjusting a few broker configs:

\n

\n
1. Set a low segment.ms (e.g., 60 s). This forces frequent segment roll‑overs, giving the cleaner a short window to operate on the previous segment.
\n
2. Keep min.cleanable.dirty.ratio at its default 0.5 or raise it if your test writes few records per segment, otherwise compaction may be postponed.
\n
3. Configure a small log.segment.bytes (e.g., 1 MiB). Smaller segments mean the cleaner rewrites less data per cycle, keeping I/O bounded to a few megabytes.
\n
4. Set a modest retention.ms (e.g., 120 s) for quick keyless‑record purge. Adjust downwards if you need faster cleanup.
\n

\n

You can monitor the cleaner’s activity through JMX: org.apache.kafka.server:type=LogCleaner counters bytesCompacted, bytesDeleted, and cleanedSegments. Sampling these before and after a write burst confirms that compaction and delete retention fire as expected.

\n\n

Suggested minimal test configuration

\n

kafka-topics.sh --create \\
  --bootstrap-server localhost:9092 \\
  --replication-factor 1 \\
  --partitions 1 \\
  --topic test-compact-delete \\
  --config cleanup.policy=compact,delete \\
  --config retention.ms=120000 \\
  --config segment.ms=60000 \\
  --config log.segment.bytes=1048576

\n

With this setup:

\n

\n
• Keyless records disappear after roughly the configured retention.ms (≈2 min).
\n
• Keyed records are compacted every 60 s if the dirty‑ratio condition is satisfied.
\n
• Cleaner I/O stays within a few megabytes per cycle because each segment is ~1 MiB.
\n

\n\n

Version note

\n

The documented behavior is consistent across Kafka 3.x and 4.x. The only version‑sensitive point is that min.cleanable.dirty.ratio was introduced in Kafka 0.10.0; newer releases keep the same default of 0.5. If you run a broker older than 0.10.0, compaction effectively never runs (ratio behaves as 1).

\n\n

What to verify next

\n

To confirm the behavior in your specific test environment, could you share the Kafka broker version you plan to use? That will help tighten the timing expectations and ensure the cleanup.policy interaction matches the documented logic.