Moleculer Service Broker and NATS Streaming Transporter: Handling In‑Flight Requests During a Failed Upgrade

Answer the question first

1. Drain before stopping: Implement a custom shutdown routine that stops accepting new calls, tracks the number of active action promises, and waits (with a hard timeout) for that counter to reach zero before invoking broker.stop().

2. Pre‑stop hook vs. transporter acks: A pre‑stop hook is required; transporter acknowledgements only protect the message queue, not the already‑executed local calls. Both mechanisms should coexist.

3. Verify redelivery: After restarting the previous version, check the NATS/JetStream consumer’s pending and redelivered metrics. A non‑zero redelivered count confirms that unacknowledged messages were re‑queued and processed.

Why a pre‑stop hook is required

• Moleculer’s broker.stop() terminates the node immediately; any action that has already entered the handler is cancelled.
• Transporters that support acknowledgements (NATS Streaming or JetStream) only guarantee that a message that was not acknowledged will be delivered to another subscriber. They cannot recover a request that has already been handed to the service logic.
• Without draining, a client that sent a request just before the upgrade may see a timeout, retry, or silently lose the result.

How to implement the drain routine

1. Set a flag (e.g., this.accepting = true) and expose an activeRequests counter via a health endpoint.
2. Wrap every action with a counter increment/decrement:

   async action(ctx) {
     this.activeRequests++;
     try {
       return await ctx.call(this.actionName, ctx.params);
     } finally {
       this.activeRequests--;
     }
   }

3. On process shutdown (SIGTERM or a custom preStop method), set this.accepting = false, then wait until this.activeRequests becomes zero or a timeout (e.g., 30 s) elapses.
4. After the wait, call await broker.stop() (or broker.stopService() for a single service).

Transporter acks vs. draining

• Acking ensures that if the node dies before the message is processed, the message is re‑queued. It does not protect a request that has already started.
• Draining guarantees that all requests that have entered the service logic finish before the node shuts down.
• Use both: the transporter handles unacknowledged messages, the drain routine handles in‑flight calls.

Verifying that unacknowledged messages are redelivered

1. During a test, send a batch of actions that deliberately delay (e.g., 5 s).
2. Immediately send SIGTERM to the upgraded node.
3. Restart the previous version.
4. Query the JetStream consumer (e.g., js.ConsumerInfo()) and confirm:
   • pending > 0 during the stop.
   • After restart, redelivered has increased.
5. Check client logs for RequestTimeoutError or retry entries – they indicate that the request was retried or redelivered, not silently dropped.

Missing diagnostic detail

To tailor the timeout and idempotency strategy, please share: Which NATS transporter (core, STAN, or JetStream) and version are you using?

Assumptions & uncertainties

• We assume the deployment uses a durable transporter (NATS Streaming or JetStream). Plain NATS does not persist unacknowledged messages.
• The timeout in the drain routine should be tuned to the longest expected action duration in your service.
• Make sure service actions are idempotent to avoid duplicate work if a call is retried after a rollback.