Managing State in Kafka: When to Use Log Compaction Over Retention
Learn how to use Apache Kafka Log Compaction to manage stateful data, handle deletions with tombstones, and avoid the pitfalls of infinite log growth.
Loading…
ReadMeFeed / Community knowledge
Apache Kafka
Real questions. Useful conversations. Find the people who know your stack.
Apache Kafka
Developer tools · #apachekafka
Managing State in Kafka: When to Use Log Compaction Over Deletion
Learn how Apache Kafka Log Compaction manages stateful data by retaining the latest value for each key, reducing storage overhead, and implementing tombstones for deletions.
Choosing Kafka Delivery Semantics: Idempotent vs. Transactional Exactly-Once
Decide between idempotent and transactional delivery in Kafka. Compare latency, operational complexity, and configurations for financial event processing.
Apache Kafka Log Compaction: Stateful Streams with Compaction
Learn how to use Apache Kafka Log Compaction to transform topics into durable key-value stores for efficient application state recovery and management.
Choosing a Kafka Producer Acknowledgment Strategy for Data Durability
Learn how to select the right Kafka producer 'acks' setting to balance throughput and durability, and why 'acks=all' requires 'min.insync.replicas' to be effective.
Apache Kafka broker resource overhead in low-traffic environments
Optimizing a Kafka deployment for low-traffic workloads requires balancing infrastructure costs against operational stability. When reducing the broker footprint to minimize CPU and memory usage, certain configuration defaults may lead to unnecessary resource consumption. Specifically, the allocation of num.network.threads and num.io.threads is typically tun
Idempotent Producer deduplication state after producer restart
Apache Kafka's idempotent producer uses a Producer ID (PID) and sequence numbers to prevent duplicate messages during retry cycles. This mechanism ensures that if a broker fails to acknowledge a write but the message was actually persisted, subsequent retries are discarded by the broker. The deduplication state is tied to the lifecycle of the producer sessio
NotEnoughReplicasException during acks=all production deployment
Producer Failure in High-Durability Configurations In a production Kafka cluster (version 3.x), a producer is configured with acks=all to ensure maximum data durability. While this configuration functions without issue in a local single-node environment, it triggers a NotEnoughReplicasException when deployed to a multi-broker production cluster. The cluster