Idempotent Producer: Limits on In‑Flight Requests and Retry Behavior

Answer

For an idempotent Kafka producer the setting max.in.flight.requests.per.connection must be ≤ 5; otherwise ordering guarantees are lost and, in Kafka 3.0+, the broker rejects the configuration with an error.

Confirmed facts

• The broker deduplicates records using the tuple (producer‑ID, epoch, sequence number) stored in the partition log.
• Only complete record batches are made durable; an incomplete batch caused by a crash is truncated on recovery, so no partial write is visible to consumers.
• When a network failure triggers a retry, the producer resends the same PID/epoch/sequence; the broker recognizes it as a duplicate and ignores it.
• For non‑transactional idempotent producers, a group rebalance does not fence in‑flight requests; they either complete or fail normally and do not create duplicates.
• For transactional idempotent producers (when transactional.id is set), the broker increments the producer epoch during a rebalance, causing any in‑flight request from the old epoch to be rejected with INVALID_PRODUCER_EPOCH.
• Kafka 3.0+ enforces the ≤ 5 limit when enable.idempotence=true (KIP‑360) and strengthens epoch fencing and producer state snapshots (KIP‑447).

Likely explanation of the concern

The worry that a broker could retain a partially written record and later cause a duplicate after a producer reconnect stems from a misunderstanding of the log append semantics. Because the broker only commits whole batches, any incomplete data is removed on restart, and the idempotency check uses the producer’s sequence number, so a retry with the same sequence is treated as a duplicate and discarded.

Steps to verify your setup

1. Confirm broker version (≥ 2.0 for default limit 5, ≥ 3.0 for explicit rejection).
2. Check producer config: enable.idempotence=true, max.in.flight.requests.per.connection ≤ 5, and appropriate acks (≥ 1) and retries > 0.
3. Inspect broker logs for messages like “Appending batch with sequence” and “Duplicate sequence” to see deduplication in action.
4. If using transactional.id, verify that transactional.id.expiration.ms is set as desired and that the producer handles INVALID_PRODUCER_EPOCH gracefully.

Missing diagnostic detail

Are you using a transactional ID (i.e., transactional.id set) or relying solely on enable.idempotence=true? The answer changes whether rebalance‑induced epoch fencing applies and whether you need to handle INVALID_PRODUCER_EPOCH.