JetStream Persistence & Consumer Models: Choosing the Right Delivery Pattern for Reliable Messaging

Problem: How to guarantee message delivery with NATS JetStream?

In many distributed systems, a single message can trigger critical downstream actions. When those actions fail, you need a way to retry or recover without losing data. NATS core offers lightning‑fast publish/subscribe, but it does not persist messages. JetStream adds persistence, yet it presents several consumer models and delivery semantics that can be confusing for developers deciding on a reliable architecture.

Thesis: Use JetStream’s pull‑based consumers with appropriate AckWait and delivery policy to balance reliability, latency, and resource usage.

Pull consumers give the client full control over when to fetch messages, allowing back‑pressure handling and efficient batching. Coupled with a well‑chosen AckWait and the right delivery policy, they provide a predictable, at‑least‑once guarantee while keeping storage usage in check.

1. JetStream Persistence Basics

JetStream extends NATS by adding a stream abstraction that stores messages on disk or memory. A stream is identified by a name and a set of subjects it consumes. Example stream definition (YAML style):

nats stream add --config \
  "\n  name: ORDERS\n  subjects: [\"orders.*\"]\n  storage: file\n  retention: limits\n  max_msgs: 100000\n  max_age: 24h\n\"

After creating the stream, you can publish messages normally:

nats pub orders.new "{\"id\": 123, \"amount\": 45.67}"

JetStream stores them until the retention policy expires. To verify persistence, restart the server:

nats server -js
# After restart
nats stream info ORDERS

A non‑empty messages count confirms persistence. If you see zero, the stream was not persisted or the server was not started with -js.

2. Consumer Models – Push vs Pull

JetStream offers two consumer types:

• Push: The server pushes messages to a subscriber’s inbox as soon as they arrive.
• Pull: The client requests a batch of messages on demand.

Push is simple but gives the server full control, which can lead to overload if the consumer is slow. Pull allows the client to apply back‑pressure and request only what it can handle.

Example pull consumer creation:

nats consumer add ORDERS my-pull-consumer --config \
  "\n  durable: my-pull-consumer\n  ack_policy: explicit\n  ack_wait: 30s\n  deliver_policy: all\n  filter_subject: orders.new\n  max_ack_pending: 100\n  replay_policy: instant\n\"

Now fetch a batch of 10 messages:

nats pull ORDERS my-pull-consumer -n 10
# JSON output contains an array of messages with IDs, payloads, and timestamps

After processing, acknowledge each message:

nats ack ORDERS my-pull-consumer --ids <msg-id-1> <msg-id-2> ...

In a Go client, you would use the JetStream API’s Consumer.Pull method, iterate over the batch, process, and call Ack on each.

3. Delivery Semantics & AckWait

JetStream supports three delivery semantics:

• At‑most‑once – Messages are delivered once; no retries.
• At‑least‑once – Messages are retried until acknowledged.
• Exactly‑once – Requires a unique MsgID and deduplication; messages are processed once.

The ack_policy field controls this. For most use‑cases, explicit (at‑least‑once) is the safest default. The AckWait setting determines how long the server waits for an ack before redelivering. A too‑short value (e.g., 1s) can cause “poison pill” loops if the consumer cannot keep up. A too‑long value (e.g., 1h) delays failure detection. A typical safe value is 30–60s for batch processing.

Example: Setting ack_wait: 45s in the consumer config. To monitor redelivery attempts, query the consumer’s status:

nats consumer info ORDERS my-pull-consumer
# Look at the "redelivered" count

4. Trade‑offs & Limitations

• Disk I/O vs Latency: Disk‑backed streams provide durability but add I/O latency compared to in‑memory. For ultra‑low latency, keep the stream in memory but be aware of data loss on crash.
• Batch Size vs Overhead: Pull consumers with tiny batch sizes (<5) increase round‑trips and network overhead. A batch of 20–50 messages is usually a good compromise.
• Redelivery Loops: Misconfigured AckWait or no ack at all can lead to endless redelivery. Always implement a max retry counter or use the built‑in max_deliver consumer option.
• Retention Policy Impact: Interest retention keeps messages only while a consumer is active. If you need long‑term storage, use limits or workqueue.

Actionable Closing

To implement a reliable message pipeline with JetStream:

1. Create a persistent stream with storage: file and a suitable retention policy.
2. Define a pull consumer with ack_policy: explicit, ack_wait: 45s, and a batch size that matches your processing throughput.
3. In your consumer code, fetch messages in batches, process them, and acknowledge each. Handle errors by not acknowledging and allowing redelivery.
4. Periodically monitor consumer status (nats consumer info) to detect stuck messages or high redelivery counts.
5. For high availability, consider stream mirroring or sourcing to replicate data across clusters.

By following this pattern, you gain durable, at‑least‑once delivery with controlled latency and resource usage. Adjust AckWait and batch size based on real‑world load, and keep an eye on the consumer’s redelivery metrics to avoid poison pill scenarios.