Apache Kafka Log Compaction: Stateful Streams with Compaction

The Problem: Stateful Recovery Without Replay

Building a stateful service—such as a user session tracker, a real-time inventory system, or a cache—requires handling service restarts or scaling events. Using a standard Kafka topic with time-based retention means replaying every event since the beginning of the retention window to rebuild the current state. As your data grows, this leads to massive recovery times and inefficient resource usage.

Transforming Kafka into a Key-Value Store

Log Compaction in Apache Kafka solves this by treating topics as durable key-value stores. Kafka retains at least the last known value for each message key, allowing applications to bootstrap their state by reading only the most recent update for each entity. This is achieved through the cleanup.policy=compact configuration, which enables the Log Cleaner process.

How Log Compaction Works

The Log Cleaner is a background process that asynchronously scans the log and discards older records that share a key with a more recent record. This process ensures that the log retains only the latest value for each key, significantly reducing storage overhead compared to time-based retention.

Handling Deletions with Tombstones

To explicitly remove a key from a compacted topic, produce a Tombstone: a message with the target key and a null value. The Log Cleaner treats the tombstone as the latest value, and it is eventually purged based on the delete.retention.ms configuration.

Practical Example: Configuring and Verifying a Compacted Topic

To implement Log Compaction, configure the topic with the appropriate cleanup policy. This example assumes Kafka 3.x and the use of the kafka-topics.sh utility.

1. Create the Compacted Topic

# Run this on the Kafka client/broker with administrative permissions
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic user-profiles \
  --partitions 3 \
  --replication-factor 1 \
  --config cleanup.policy=compact

2. Produce State Updates

Produce several updates for the same key to simulate state changes:

# Produce initial state
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic user-profiles --property "parse.key=true" --property "key.separator=\" \
  "user1:Active" \
  "user2:Active" \
  "user1:Away"

3. Verify the Result

After the Log Cleaner cycle runs (which can take several minutes depending on broker settings), consume the topic from the beginning:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-profiles --from-beginning --property "print.key=true"

Expected Result: You should see user1:Away and user2:Active. The initial user1:Active record will have been compacted away.

Trade-offs and Performance Considerations

While Log Compaction is powerful, it comes with specific trade-offs:

• Disk I/O Spikes: The Log Cleaner can cause significant disk I/O pressure on topics with high key cardinality and frequent updates, potentially impacting producer latency.
• Read Latency: Although compaction reduces the total amount of data to read during recovery, it does not provide the point-lookup performance of a database. You still have to scan the log to find a specific key.
• Hybrid Retention: You can combine compaction with deletion policies (cleanup.policy=compact,delete) to ensure that no record is kept longer than a specific time, useful for compliance with regulations like GDPR.

Summary and Verification Checklist

To ensure your compaction strategy is working as intended, check the following:

• Confirm cleanup.policy is set to compact via kafka-configs.sh --describe.
• Verify that delete.retention.ms is long enough for your slowest consumer to process tombstone records.
• Monitor the LogCleaner metrics in JMX to ensure the cleaner is keeping up with the production rate.