Dual‑write vs rolling upgrade for zero‑downtime NATS consumer migration

Answer

Use consumer lag = 0 and pending acknowledgments = 0 as the observable trigger to switch the durable consumer exclusively to the new JetStream stream.

Likely explanation

Dual‑write lets old and new consumer versions run side‑by‑side, both reading the same subject and writing results downstream. This avoids stopping consumers but can cause duplicate processing unless the downstream logic is idempotent or can deduplicate using the NATS message ID.

Confirmed fact

A rolling upgrade (blue‑green/canary) upgrades consumer instances one at a time while keeping the rest of the fleet on the old version. Only one version processes any given message, eliminating duplicates, but each instance must be stopped and started briefly.

Steps for this case (rolling upgrade preferred when duplicates are unsafe)

1. Pause publishers (or verify they are idle).
2. Monitor the current durable consumer: ensure NumPending = 0 and AckPending = 0 via nats jetstream consumer info or a monitoring dashboard.
3. Wait a quiescent interval of at least one ack_wait after the last acknowledgment (see details below).
4. Stop one consumer instance gracefully.
5. Start the new version binary with the identical durable consumer name and subscription configuration.
6. Wait for the instance to catch up (lag ≈ 0) and process messages without errors.
7. Return the instance to the pool.
8. Repeat for each remaining instance until all run the new version.
9. Resume publishers.

Verification that no messages are missed or duplicated

• Before stopping an old instance, confirm AckPending = 0 and NumPending = 0; this guarantees all delivered messages have been acknowledged.
• After the new instance starts, check that its Delivered sequence starts exactly where the old instance left off (no gap) and that NumPending trends to zero.
• Query the downstream store for duplicate NATS message IDs; absence of duplicates confirms idempotent handling or a clean cut‑over.
• After migration, verify all running processes report the new version binary (e.g., via ps or a version endpoint) and that no old version instances remain.

Minimum quiescent interval

After publishers pause, wait for at least one ack_wait period after the last acknowledgment. If the consumer is configured with max_pending > 0, a safer bound is ack_wait × (max_pending + 1) to ensure any in‑flight messages have time to be acknowledged and removed from the pending queue.

Missing diagnostic detail that could change the recommendation

Determine whether the consumer’s processing is idempotent or can safely deduplicate using the NATS message ID. If duplicates would cause incorrect state, a rolling upgrade is preferred; otherwise dual‑write may be used for a faster cut‑over.