Designing a Kafka Topic with Log Compaction: Requirements, Minimal Setup, Boundaries, Checks, and Failure Modes

Requirements

Log compaction is used when a topic stores a changelog of entity state and you need the most recent value per key while discarding older duplicates. Log compaction retains the latest offset for each key, enabling infinite retention of change logs while discarding older duplicates.

Operational requirements:

• Disk space for at least two copies of the largest segment during merge. Compaction may increase disk I/O and temporary storage usage during segment merging, requiring capacity planning.
• I/O headroom for the compaction thread.
• Monitoring for JMX metric compaction.lag and segment size growth.

Smallest Suitable Design

Minimal configuration:

1. Create topic with cleanup.policy=compact or compact,delete for a time safety net.
2. Set segment.bytes to a reasonable size, e.g., 256 MiB.
3. Leave retention.ms at Long.MAX_VALUE. The feature interacts with topic retention.ms and retention.bytes; compaction does not reduce log size below the minimum retention window.
4. Use min.cleanable.dirty.ratio default 0.5 to trigger compaction when enough duplicates exist.

Example creation as Kafka admin:

kafka-topics.sh --create --bootstrap-server broker1:9092 --topic user-profile-changelog --partitions 6 --replication-factor 3 --config cleanup.policy=compact --config segment.bytes=268435456 --config min.cleanable.dirty.ratio=0.5

Trust/Data Boundaries

Compaction runs inside the broker log directories. Trust boundary is the broker filesystem and JVM process.

• log.dirs must be writable by the Kafka OS user.
• Backups and replication protect against disk loss, as compacted segments permanently remove older duplicates.
• Network security protects producer/consumer traffic; compaction does not expose data.

Operational Checks

Verify compaction progress:

1. JMX metric kafka.log:type=LogCompactionManager,name=CompactionLag per topic-partition. Use JMX metric 'compaction.lag' to monitor progress of compaction tasks.
2. Run describe tool: Run 'kafka-log-compaction.sh --describe ' to view compaction status and last offset.

kafka-log-compaction.sh --describe --bootstrap-server broker1:9092 --topic user-profile-changelog

Investigate if lag does not advance: high-cardinality keys, insufficient min.cleanable.dirty.ratio, or disk I/O saturation.

Failure Modes

• Latency spikes while segments are rewritten. Compaction can cause temporary latency spikes as segments are rewritten.
• Temporary storage growth during merge can trigger OutOfSpaceException.
• Ineffective compaction if retention.bytes or retention.ms is too low.
• High-cardinality key explosion. High-cardinality keys may lead to excessive log growth if not partitioned appropriately.

Conditions That Would Change the Design

1. Key cardinality grows beyond partition capacity.
2. Tighter latency budgets require disabling compaction.
3. Hard upper bound on log size needed, add retention.bytes with compaction.
4. Persistent compaction lag may indicate cleaner stalls; check Kafka version.

Limitations

Compaction is not immediate; window depends on segment size and cleaner throttle. Misconfigured retention settings may result in compaction being ineffective or delayed. Tombstones require delete retention to be purged eventually.