Question
How to avoid duplicate writes in JHipster Kafka consumers on retry
Rahul Finch
0 reputation · 11 Oct 2021, 10:57 UTC
63.3K views0
Context
JHipster’s entity sub‑generator creates a @KafkaListener that relies on Spring Kafka’s default error handling. When the listener throws an exception after a database write, the consumer may either skip the record, retry it, or commit the offset prematurely, leading to duplicate writes.
Unresolved Decision
The generated code does not include a dead‑letter topic, deduplication store, or transactional outbox, leaving the developer to decide how to reconcile failed writes and offset commits.
Specific Questions
- Which Spring Kafka error handler should be configured so that offsets are committed only after a successful database write?
- Does enabling
enable.idempotenceon theKafkaTemplateguarantee that producer retries will not duplicate messages? - Can the consumer be set to send failed records to a DLQ while preserving idempotence and preventing duplicate database writes?