Question
Dual‑write vs rolling upgrade for zero‑downtime NATS consumer migration
Neel Orbit
0 reputation · 19 Apr 2023, 14:24 UTC
69.7K views
Loading…
0 reputation · 19 Apr 2023, 14:24 UTC
Goal: achieve zero‑downtime migration of a small application that publishes to a JetStream stream, moving consumers from an existing stream to a new stream while preserving exactly‑once delivery.
Constraints: must keep ack wait and max pending compatible with both streams, avoid message loss or duplication, and rely on documented NATS features (dual‑write or rolling server upgrade) without introducing custom logic that is not verified.
Uncertainty: the NATS documentation does not define a precise checkpoint or signal (e.g., lag, pending acks, quiescent period) for safely rebinding a durable consumer from the old to the new stream, leaving the switchover timing to implementer‑defined logic.
What observable metric (such as consumer lag, pending acknowledgments, or a period of zero publish activity) can be used as a safe trigger to rebind the consumer exclusively to the new stream?
How can we verify that no messages are missed or duplicated during the instant when the consumer binding switches from the old stream to the new stream?
Is there a recommended minimum quiescent interval after publishers pause before performing the rebind, and how should it be determined based on message rate and ack wait?
32775 reputation · 19 Apr 2023, 18:37 UTC
Use consumer lag = 0 and pending acknowledgments = 0 as the observable trigger to switch the durable consumer exclusively to the new JetStream stream.
Dual‑write lets old and new consumer versions run side‑by‑side, both reading the same subject and writing results downstream. This avoids stopping consumers but can cause duplicate processing unless the downstream logic is idempotent or can deduplicate using the NATS message ID.
A rolling upgrade (blue‑green/canary) upgrades consumer instances one at a time while keeping the rest of the fleet on the old version. Only one version processes any given message, eliminating duplicates, but each instance must be stopped and started briefly.
natsNumPending = 0 and AckPending = 0 via nats jetstream consumer info or a monitoring dashboard.ack_wait after the last acknowledgment (see details below).lag ≈ 0) and process messages without errors.AckPending = 0 and NumPending = 0; this guarantees all delivered messages have been acknowledged.Delivered sequence starts exactly where the old instance left off (no gap) and that NumPending trends to zero.ps or a version endpoint) and that no old version instances remain.After publishers pause, wait for at least one ack_wait period after the last acknowledgment. If the consumer is configured with max_pending > 0, a safer bound is ack_wait × (max_pending + 1) to ensure any in‑flight messages have time to be acknowledged and removed from the pending queue.
Determine whether the consumer’s processing is idempotent or can safely deduplicate using the NATS message ID. If duplicates would cause incorrect state, a rolling upgrade is preferred; otherwise dual‑write may be used for a faster cut‑over.
Use comments to ask for clarification. Post a solution as an answer.
32,775 reputation · 19 Apr 2023, 20:03 UTC
Besides checking that NumPending and AckPending are zero, you must also wait for at least one full ack_wait interval after the last acknowledgment to ensure no redeliveries are still in flight. JetStream’s consumer info exposes Delivered and AckFloor; when Delivered - AckFloor equals zero and remains so for a duration ≥ ack_wait, the consumer is quiescent and safe to stop or replace.