Fusion 5.x Index Pipeline Retries Require Explicit Idempotency Keys for Deduplication

Answer

Yes – Fusion 5.x should offer an optional pipeline‑level setting that automatically generates a per‑document idempotency key, but only if the key is derived from immutable data before any non‑deterministic enrichment stages run.

Confirmed Facts

• Fusion 5.x index pipelines support configurable retry policies (exponential back‑off, max attempts) on stages such as the Solr Indexer.
• The platform does not expose a pipeline‑wide idempotencyKey property; deduplication today relies on a stable document id field that Solr treats as an upsert when overwrite=true.
• If a stage mutates content non‑deterministically (e.g., adds a timestamp or random UUID) while preserving the same id, a retry will re‑index the altered document, causing the new content to overwrite the previous version rather than being filtered out.

Likely Explanation

An automatic idempotency key would need to be computed from fields that are guaranteed not to change during processing (e.g., source‑system primary key, immutable metadata). By generating the key before any enrichment stage, Fusion can attach it as metadata or as the document identifier, allowing the target store to recognize retries and skip duplicate writes.

Steps to Implement This Case

1. Identify a set of immutable fields (e.g., sourceId, eventTimestamp if it never changes).
2. Add a pre‑enrichment JavaScript (or Scala) stage that computes a hash (SHA‑256, MD5, etc.) of those fields and stores the result in a new field such as _idempotencyKey.
3. Ensure this stage runs before any non‑deterministic enrichment (timestamp, UUID, random sampling, etc.).
4. For Solr targets: configure the Solr Indexer stage to use _idempotencyKey as the document id (or add it as a regular field and keep overwrite=true).
5. For non‑Solr targets that lack native upsert: pass the _idempotencyKey as a header or metadata field and implement duplicate detection on the target side (e.g., a cache of seen keys or a conditional write).

Missing Diagnostic Detail

Does the pipeline contain any non‑deterministic enrichment stages that execute after the document id is set? If yes, the idempotency key must be generated prior to those stages; if no, relying on the existing stable id may be sufficient.