Log compaction with cleanup.policy compact,delete and keyless message handling
0 reputation · 24 Mar 2020, 16:57 UTC
Design goal is to configure a Kafka topic for an integration test that uses log compaction together with time-based deletion, without access to production credentials.
The topic should retain only the latest value per key under compaction, while messages without keys should be subject to delete retention. The cleanup.policy is intended to be compact,delete and min.cleanable.dirty.ratio is left at its default.
Uncertainty remains about the permission boundary for keyless records under a combined policy and about the timing and resource impact of compaction under sustained write load. The interaction with segment.ms is also not strictly guaranteed.
What is the documented retention behavior for keyless records when cleanup.policy is compact,delete? Under combined policy, does compaction precedence affect delete retention for keyless messages? How should the test configuration bound expected compaction timing and I/O impact without production metrics?