Fusion 5.x Index Pipeline Retries Require Explicit Idempotency Keys for Deduplication

Background

Fusion 5.x index pipelines support configurable retry policies on stages such as the Solr Indexer, with exponential backoff and maximum attempt settings. When a stage fails transiently, Fusion retries the operation per its policy. Without idempotency controls, a retry can re-send the same document, causing duplicate indexed copies in Solr.

Current Behavior

The documented mitigation relies on a stable document id field so that Solr's upsert semantics (add with overwrite=true) prevent duplication on retry. The idempotencyKey concept is not a first-class pipeline-wide setting in 5.x; it is typically implemented via the stable id or a custom JavaScript stage. If a pipeline stage mutates document content non-deterministically (for example, timestamp enrichment or random UUID generation), retries with the same ID will overwrite with different content rather than deduplicate.

Open Decision

An unresolved design question in Fusion 5.x is whether the platform should automatically generate and track idempotency keys for pipeline executions (for example, a per-document hash) or continue requiring explicit user configuration, leaving duplicate risk to implementation choice. This boundary affects upgrade paths from Fusion 4.x, where default retry configurations differed, and extends to non-Solr index targets that may not provide upsert semantics by default.

Should Fusion 5.x introduce automatic per-document idempotency key generation as a pipeline-level option? How would automatic keys interact with non-deterministic enrichment stages? What compatibility guarantees are needed for custom index targets that lack native upsert behavior?