Solving Message Loss in NATS: Moving from Core Pub/Sub to JetStream

The 'Fire and Forget' Problem

Standard NATS core is a high-performance messaging system, but it operates on a "fire and forget" model. If a subscriber is offline when a message is published, that message is gone forever. For telemetry or real-time dashboards, this is fine. For order processing, billing, or state synchronization, it is a critical failure.

The solution is JetStream, the persistence layer for NATS. JetStream transforms NATS from a simple message bus into a distributed log, allowing you to store messages on disk and ensure they are delivered at least once, regardless of whether the consumer was online at the time of publication.

How JetStream Ensures Delivery

JetStream introduces the concept of a Stream. While a standard NATS subject is just a routing address, a Stream is a configuration that tells the NATS server: "Any message sent to these subjects should be written to storage."

To guarantee delivery, JetStream relies on Acknowledgements (ACKs). When a consumer receives a message, the server marks it as "delivered" but not "processed." The server waits for an ACK from the client. If the client crashes or the network drops before the ACK is sent, the server redelivers the message after a predefined timeout.

Pull vs. Push Consumers

Choosing the right consumer model is the most important architectural decision when using JetStream:

• Push Consumers: The server pushes messages to the client as they arrive. This is lower latency but can easily overwhelm a slow consumer (the "firehose" effect).
• Pull Consumers: The client explicitly requests a batch of messages (e.g., "Give me 10 messages"). This provides built-in flow control, as the client only asks for what it can handle.

Implementation: Creating a Durable Pipeline

To implement a durable pipeline, you must first enable JetStream on the server using the -js flag during startup. Once enabled, you can use the NATS CLI to configure your infrastructure.

Step 1: Define the Stream

Run this command on your local machine with the NATS CLI installed. This creates a stream named ORDERS that captures all messages on subjects starting with orders.

nats stream add ORDERS --subjects "orders.*" --storage file --retention limits

Note: --storage file ensures messages survive a server restart. --retention limits means messages are kept until the stream reaches a size or age limit.

Step 2: Create a Durable Pull Consumer

A "Durable" consumer is one where the server remembers the last acknowledged message (the offset) even if the client disconnects.

nats consumer add ORDERS ORDER_PROCESSOR --pull --deliver-all --ack-explicit

Required Permissions: The user executing these commands must have administrative permissions to create streams and consumers on the NATS cluster.

Step 3: Verify Redelivery

To test the "at-least-once" guarantee, follow this sequence:

1. Publish a message: nats pub orders.new "Order #123"
2. Start a consumer that pulls the message but does not send an ACK.
3. Kill the consumer process.
4. Restart the consumer. You will see the same message redelivered because the server never received the ACK.

Trade-offs and Constraints

JetStream is not a free upgrade; it introduces specific engineering costs:

Metric	Core NATS	NATS JetStream
Latency	Ultra-low (Memory)	Higher (Disk I/O)
Reliability	At-most-once	At-least-once
Complexity	Zero config	Requires Stream/Consumer management

The Poison Pill Risk: If a message is malformed and causes your consumer to crash every time it is processed, JetStream will redeliver it indefinitely. To prevent this, always configure a max_deliver limit on your consumer. This ensures that after X failed attempts, the message is moved to a Dead Letter Office or dropped.

Closing Checklist

When moving to JetStream, verify these three items to avoid production outages:

• Disk Space: Monitor the volume where NATS stores data; if the disk fills up, the stream will stop accepting new messages.
• ACK Wait: Ensure your ack_wait timeout is longer than the maximum time your application takes to process a single message.
• Consumer Names: Use consistent durable names for your consumers so that offsets are tracked correctly across pod restarts in Kubernetes.