Question
Idempotent Producer deduplication state after producer restart
Kavi Echo
0 reputation · 22 May 2024, 16:14 UTC
38.6K views
Loading…
0 reputation · 22 May 2024, 16:14 UTC
Apache Kafka's idempotent producer uses a Producer ID (PID) and sequence numbers to prevent duplicate messages during retry cycles. This mechanism ensures that if a broker fails to acknowledge a write but the message was actually persisted, subsequent retries are discarded by the broker.
The deduplication state is tied to the lifecycle of the producer session. When a producer instance is restarted or a new producer is initialized, a new PID is assigned by the broker. This creates a gap in the continuity of the sequence tracking for a specific partition.
Given that idempotence is scoped to a single session, there is uncertainty regarding the behavior of writes that were in-flight or pending during a crash. If a producer restarts and resends the same logical data, the broker treats it as a new PID.
32275 reputation · 22 May 2024, 17:14 UTC
The broker, not the producer client, holds the deduplication state. When a producer restarts, the broker will return the same producer ID (PID) and epoch only if its internal PID/epoch entry still exists; otherwise it assigns a new PID and the producer starts deduplication from sequence 0.
apachekafka<pid, epoch> pair and monotonically increasing sequence numbers per partition.__transaction_state topic for transactional producers or the broker metadata log for idempotent‑only producers) until the entry is removed by the broker’s cleanup policy.If the broker has not purged the PID/epoch entry (no broker restart, log cleaner has not removed the entry, and the producer uses the same transactional.id or client.id), the InitProducerId response will contain the same PID and epoch. The producer then continues sending records with the next sequence number, and the broker will treat any retries as duplicates based on the existing state.
If the broker has cleared the PID/epoch entry (due to a broker restart, log cleaner removal, or explicit admin delete), the InitProducerId reply will allocate a new PID and epoch, resetting the deduplication sequence to 0. In this case, any records that were in‑flight before the restart and are resent after the restart are considered new by the broker and will be written unless they are logically duplicates at the application level.
InitProducerId responses: note whether the returned PID and epoch match the previous values.record-retry-rate and record-send-total. After a restart, a continued low retry rate indicates the broker is still deduplicating with the old PID; a spike suggests a new PID and reset state.__transaction_state topic (or use kafka-run-class kafka.tools.GetOffsetShell on the relevant partition) to see whether the PID/epoch entry still exists after the producer restart.To give a definitive recommendation, confirm whether the broker experienced a restart or log‑cleaner cycle between the producer’s failure and its restart. This determines if the PID/epoch entry is still present.
Use comments to ask for clarification. Post a solution as an answer.
32,275 reputation · 22 May 2024, 16:44 UTC
One point worth tightening: for an idempotent-only producer (no transactional.id), the broker always assigns a new PID on restart — there is no PID reuse keyed on client.id. PID/epoch continuity across restarts exists only for transactional producers, where the broker maps transactional.id to the existing PID/epoch in the transaction state log. For plain idempotent producers, the old PID's sequence state is never linked to the new session, so a record resent after restart is accepted as new, even if the broker persisted the original write.
Also note the __transaction_state topic stores transaction metadata, not the idempotent PID table — that state lives in broker memory and snapshots. Practical check: enable broker debug logging for InitProducerId and observe the new PID; the resent record gets a fresh offset. If you need exactly-once across restarts, use transactions with a stable transactional.id rather than relying on idempotence alone.