Reliable Microservice Messaging with NATS JetStream Durable Consumers

The problem: lost work when a consumer restarts

In a typical microservice setup a NATS subscriber processes messages and acknowledges them. If the process crashes or is redeployed, the client loses its in‑memory cursor. When it reconnects it either starts from the beginning of the stream (re‑processing everything) or from the latest message (missing the ones it never acked). Both outcomes break exactly‑once guarantees and waste CPU.

How durable consumers fix it

JetStream stores the acknowledgment position on the server. A durable consumer is a named subscription that survives client disconnects. When the client reconnects with the same durable name, JetStream resumes delivery at the last acknowledged message, guaranteeing at‑least‑once delivery without manual offset management.

Creating a stream and a durable consumer

Assume you have a NATS server running with JetStream enabled (nats-server -js). The following steps are run from a machine that has the nats CLI installed and network access to the server (default port 4222). You need permission to create streams and consumers (usually the admin or a scoped JetStream account).

1. Create a stream that retains messages for a day:

   nats stream add ORDERS --subjects 'orders.>' --retention limits --max-age 24h --storage file

2. Add a durable consumer with explicit acknowledgment:

   nats consumer add ORDERS --durable order-worker --ack explicit --deliver all

3. Publish a few test messages:

   nats pub orders.new '{"id":1}' 
   nats pub orders.new '{"id":2}'
   nats pub orders.new '{"id":3}'

4. Start a consumer process (any language) that connects as durable order-worker, processes each message, and calls msg.Ack(). After the first two messages are acked, kill the process.
5. Restart the same consumer. It will receive only message 3, because the server’s ack floor for order-worker is now at sequence 2.

Choosing an acknowledgment mode

Mode	Behavior	Typical use case
explicit	Client must call Ack() for each message.	Work that must not be lost; you can control exactly when a message is considered done.
auto	Server acknowledges as soon as the message is delivered.	High‑throughput fire‑and‑forget pipelines where occasional loss is acceptable.
none	No acknowledgment; server never redelivers.	Idempotent event sourcing where the consumer tracks its own cursor.

Operational trade‑offs

• Resource consumption: Each durable consumer keeps an ack state entry on every JetStream node. Hundreds of consumers on a large stream can increase memory and disk usage noticeably.
• Duplicate delivery: If a client crashes after processing but before sending Ack(), JetStream will redeliver the message. Your handler must be idempotent (e.g., upsert by primary key) or you must implement a deduplication layer.
• Cluster replication latency: In a multi‑node JetStream cluster the ack state is replicated via Raft. Under heavy load the ack floor may lag a few milliseconds, which is usually harmless but worth measuring if you need strict ordering.

Verifying the consumer state

Run the following to see the current ack floor and pending acknowledgments:

nats consumer info ORDERS order-worker

Look for ack_floor (the highest sequence fully acknowledged) and num_ack_pending (messages delivered but not yet acked). If num_ack_pending stays high, your consumer is falling behind.

Next steps

1. Deploy the stream and durable consumer in a staging environment that mirrors your production cluster size.
2. Enable JetStream metrics (Prometheus endpoint) and alert on jetstream_consumer_ack_pending growing beyond a threshold.
3. Make every message handler idempotent; a simple pattern is “insert … on conflict do update”.
4. Periodically review the stream’s max_bytes and max_age limits to avoid unbounded growth.