Configuring Kafka Log Compaction for State Recovery in Stream Processing

Why compaction matters for state

If you rebuild application state by replaying a Kafka topic — for example, restoring a Kafka Streams KTable or warming a cache after a restart — replay time grows with every update ever written. Log compaction solves this by guaranteeing that Kafka keeps at least the most recent value for each message key in a partition, discarding older values for the same key during background cleaning. Consumers that read from the beginning still see every key's latest state, but far fewer records.

The trade-off: compaction is asynchronous and eventual. A compacted topic is not a point-in-time database, and it only works for keyed messages. This guide covers enabling it, tuning when cleaning runs, and verifying the behavior with real producers and consumers.

Prerequisites

• A running Kafka cluster (examples assume Kafka 2.8+ with KRaft or ZooKeeper; the compaction mechanics are the same).
• The Kafka CLI tools (kafka-topics.sh, kafka-configs.sh, kafka-console-producer.sh, kafka-console-consumer.sh) from the broker distribution, run from a machine with network access to the cluster.
• Permission to create and alter topics (ACLs Create and AlterConfigs if authorization is enabled).
• Producers that always set a message key. Messages with null keys are never compacted and will accumulate forever in a compact topic — this is the most common operational surprise.

Enable compaction on a topic

Create the topic with the compact policy, or alter an existing one. Run these from any host with the CLI tools and bootstrap access; no broker restart is needed.

kafka-topics.sh --bootstrap-server broker:9092 \
  --create --topic user-state \
  --partitions 6 --replication-factor 3 \
  --config cleanup.policy=compact

To convert an existing topic (safe to do online; cleaning applies to new segments going forward):

kafka-configs.sh --bootstrap-server broker:9092 \
  --alter --entity-type topics --entity-name user-state \
  --add-config cleanup.policy=compact

You can also combine policies: cleanup.policy=compact,delete compacts and still deletes segments older than retention.ms. Use this if unbounded key growth is a risk.

Tune when cleaning actually runs

Compaction is performed by the broker's Log Cleaner threads, not at write time. Three settings matter most:

For a state-recovery topic where fast restore matters more than broker I/O, a dirty ratio of 0.1–0.2 keeps the log small. Set it per topic:

kafka-configs.sh --bootstrap-server broker:9092 \
  --alter --entity-type topics --entity-name user-state \
  --add-config min.cleanable.dirty.ratio=0.2

Verify the behavior

First confirm the configuration stuck:

kafka-configs.sh --bootstrap-server broker:9092 \
  --describe --entity-type topics --entity-name user-state

Then prove semantics with the console tools. Produce several updates for one key (the key precedes the colon):

kafka-console-producer.sh --bootstrap-server broker:9092 \
  --topic user-state \
  --property parse.key=true --property key.separator=:
>user-1:{"balance":100}
>user-1:{"balance":250}
>user-1:{"balance":90}

Immediately consuming from the beginning will show all three records — compaction has not run yet. To observe cleaning without waiting for the dirty ratio on a small test, force a roll by producing enough data or temporarily lowering segment.bytes and min.cleanable.dirty.ratio on a throwaway topic. After cleaning, a consumer with --from-beginning sees only {"balance":90} for user-1.

To test deletion, send a tombstone — a null value for the key:

>user-1:null

After the next cleaning cycle plus delete.retention.ms, the key disappears entirely from a from-beginning read. Until then, consumers see the null and must interpret it as a delete (Kafka Streams does this automatically).

Recovery and rollback options

Changing cleanup.policy back to delete is safe and immediate for future retention decisions; already-compacted segments are not restored — discarded records are gone. That is the real risk boundary: never compact a topic whose full event history you still need. If you are unsure, keep a separate append-only audit topic alongside the compacted state topic.

If disk usage grows unexpectedly on a compacted topic, check for null-key messages (they are ineligible for cleaning) and for a dirty ratio that never triggers because traffic is too low — cleaning eligibility is evaluated against segment size, so a quiet topic may hold stale records longer than expected. Broker metrics under kafka.log:type=LogCleanerManager show how much of the log is cleanable.

Limitations to design around

• Compaction guarantees at least the latest value per key, not exactly one — duplicates can remain briefly, so consumers must be idempotent.
• Ordering is preserved per key, but a from-beginning read after compaction is not the original event history.
• It is not a queryable store: you can only recover state by replaying the topic, which is exactly what KTables and changelog topics rely on.

Verify behavior on your exact broker version before relying on defaults; configuration names here are stable across the 2.x and 3.x lines, but defaults are worth confirming in your deployment's documentation.