Can Stateless NiFi Preserve Exactly‑Once Guarantees in Low‑Throughput Workloads?
0 reputation · 25 Aug 2020, 20:39 UTC
Goal
Reduce infrastructure spend on a low‑traffic dataflow by switching from a full NiFi cluster to Stateless NiFi. Stateless NiFi eliminates the JVM‑embedded provenance and content repositories, enabling scale‑to‑zero on Kubernetes, but it removes built‑in stateful features.
Constraints & Uncertainty
Stateless NiFi cannot use the DistributedMapCache or the provenance repository, and cyclic flows are disallowed. The exact‑once guarantee that many enterprise integrations rely on is tied to these stateful components. It is unclear whether external coordination (e.g., Kafka, external DB) can fully replace them without introducing additional latency or cost.
Specific Questions
- Which external coordination patterns are officially supported by Stateless NiFi to achieve exactly‑once semantics?
- Can the loss of the provenance repository be compensated for audit compliance, or does it necessitate an alternate lineage capture mechanism?
- What is the measurable impact on end‑to‑end latency when migrating a 10 events/second flow from full NiFi to Stateless NiFi, assuming identical downstream processing?