Configuring Kafka Log Compaction to Keep the Latest State per Key

What you are trying to achieve

You have a topic where each message is an update to some entity — a user profile, a product price, a device reading — and consumers only care about the most recent value for each key. By default Kafka retains messages by time or size, so old updates pile up and new consumers must replay everything to rebuild current state. Log compaction fixes this: it guarantees that each partition keeps at least the latest record for every message key, while older superseded values are removed by a background cleaner.

This guide covers enabling compaction on a topic, writing keyed records and tombstones correctly, and verifying that the cleaner actually ran. It assumes Apache Kafka 2.x or 3.x with the standard bundled command-line tools.

Prerequisites

• A running Kafka cluster (KRaft or ZooKeeper mode both work) and a broker you can reach.
• The Kafka client tools (kafka-topics.sh, kafka-console-producer.sh, kafka-console-consumer.sh) on a machine with network access to the brokers.
• Permission to create or alter topics. If your cluster enforces ACLs, you need Create/AlterConfigs on the topic resource.
• A producer that sets a message key on every record. Keyless records cannot be compacted; depending on broker version and configuration, the cleaner will skip dirty segments containing them or log errors.

Enable compaction on a topic

Compaction is controlled by the cleanup.policy config. Set it at topic creation (run on any machine with the client tools; no broker login required):

kafka-topics.sh --bootstrap-server broker1:9092 \
  --create --topic customer-state \
  --partitions 6 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.2 \
  --config segment.ms=3600000

Two supporting settings matter here. min.cleanable.dirty.ratio is the ratio of un-compacted ("dirty") log bytes to total log bytes that triggers cleaning — lower values clean more aggressively at the cost of more disk I/O. segment.ms controls how often log segments roll; only inactive segments are eligible for cleaning, so a very large segment delays compaction even if the dirty ratio is exceeded.

To convert an existing topic, alter the config instead. This changes cluster state, so confirm nothing downstream depends on time-based retention of superseded records first:

kafka-topics.sh --bootstrap-server broker1:9092 \
  --alter --topic customer-state \
  --add-config cleanup.policy=compact

To roll back, remove the override with --alter --delete-config cleanup.policy, which returns the topic to the broker default (delete).

Write keyed records and tombstones

Compaction keys everything off the record key. A quick manual exercise with the console producer (keys and values separated by a tab):

kafka-console-producer.sh --bootstrap-server broker1:9092 \
  --topic customer-state \
  --property parse.key=true --property key.separator=\t
>alice\t{"plan":"free"}
>bob\t{"plan":"pro"}
>alice\t{"plan":"enterprise"}

To delete a key entirely, produce a record with that key and a null value — a "tombstone." The cleaner removes prior values for the key, keeps the tombstone for delete.retention.ms (default 24 hours) so lagging consumers still see the deletion, then removes the tombstone too. In the console producer you can send an empty value to approximate this, but real applications should use the client API's explicit null value, e.g. in Java: producer.send(new ProducerRecord<>("customer-state", "alice", null));

How the cleaner actually behaves

Compaction is not immediate and not continuous. Each broker runs log cleaner threads that periodically scan partitions, build an in-memory offset map of the latest offset per key, and rewrite eligible segments keeping only the newest record per key (plus any records in the active segment, which is never cleaned). Expect a lag between producing updates and seeing old values disappear; on a quiet topic this is often minutes, driven by segment roll and the dirty-ratio threshold.

Three practical consequences:

1. Ordering within a key is preserved, but consumers reading from the beginning may still see several versions of a key if the cleaner has not run yet. Design consumers to tolerate duplicates by taking the last value per key.
2. Disk usage is not monotonic savings. Cleaning rewrites segments, so during cleaning you temporarily need extra disk headroom, and high key churn increases cleaner I/O.
3. Compaction is not a query layer. Reading the current value of one key means scanning the partition or maintaining an external index or state store (for example a Kafka Streams KTable/RocksDB store). Do not point per-request lookups directly at a compacted topic.

Verify the result

First confirm the config stuck:

kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --topic customer-state

The output should list cleanup.policy=compact under topic configs.

Then force conditions for cleaning: produce several versions of the same keys, wait for at least one segment roll (or set segment.ms low on a test topic), and read the topic from the start:

kafka-console-consumer.sh --bootstrap-server broker1:9092 \
  --topic customer-state --from-beginning \
  --property print.key=true --property key.separator=:

Before the cleaner runs you will see every version; after a successful cleaning cycle, re-reading shows only the latest value per key (for example, only alice:{"plan":"enterprise"} from the records above). If old values persist, check in order: that every record had a key, that a segment has actually rolled (segment.ms or segment.bytes), and that the dirty ratio threshold was crossed. Broker logs contain cleaner activity lines you can grep for LogCleaner to confirm threads are running.

Limitations to plan around

Compaction retains the latest value per key, not a complete history — if you need audit history, keep a separate time-retained topic. Tombstoned keys disappear entirely after delete.retention.ms, so consumers that are offline longer than that may miss deletions. Finally, the offset map used by the cleaner is sized by log.cleaner.dedupe.buffer.size; topics with extremely many distinct keys per segment may need that raised, which costs heap on the broker. Validate behavior on a test topic with your real key cardinality before compacting production state topics.