Building a Durable JetStream Stream and Consumer in NATS 2.10+: A Practical Guide

Problem Statement

In many distributed systems you need a messaging layer that guarantees that messages survive server restarts and that consumers can replay or recover unacknowledged messages. NATS’ legacy Streaming server has been superseded by JetStream, which embeds persistence, replay, and consumer state directly into the NATS server process. This guide shows how to create a durable stream and a pull consumer that can reliably recover after a crash, using NATS 2.10 or newer.

Desired Outcome

• Define a JetStream stream that stores messages on disk (File storage).
• Configure a pull consumer that acknowledges messages and can replay from the beginning.
• Verify that messages survive a consumer crash and that the server reports no errors.
• Understand the key configuration knobs that affect durability and recovery.

Prerequisites

• Running NATS 2.10+ (download from GitHub releases).
• Access to a terminal with nats CLI installed (available via brew install nats-io/nats-tools/nats or npm i -g nats-cli).
• Sufficient disk space on the node where the server runs (at least 1 GB for the example).
• Administrative permissions to start the server and create streams/consumers.

Procedure

1. Start a NATS Server with JetStream Enabled

# Run in a dedicated terminal
nats-server -js -DV

The -js flag turns on JetStream. -DV prints detailed server logs, useful for debugging. The server listens on 4222 by default.

2. Define a Durable Stream

Streams are collections of messages that share a subject pattern. The following command creates a stream named ORDERS that captures all subjects beginning with orders.** and stores messages on disk.

nats stream add ORDERS \
  --subjects orders.** \
  --storage file \
  --replicas 1 \
  --max-msgs 1000000 \
  --max-msg-size 1024000 \
  --retention limits \
  --discard new \
  --duplicate-window 30s

Key knobs explained:

• --storage file – persistence to disk.
• --max-msgs – upper bound on stored messages.
• --retention limits – drop the oldest messages when the limit is reached.
• --duplicate-window – de‑duplication window for idempotent publishers.

3. Create a Pull Consumer

Pull consumers request messages on demand, making it easier to implement back‑pressure and manual acknowledgment.

nats consumer add ORDERS orders-consumer \
  --type pull \
  --ack-policy explicit \
  --max-deliver 10 \
  --deliver-policy all \
  --replay policy

Interpretation of flags:

• --ack-policy explicit – the consumer must acknowledge each message.
• --max-deliver 10 – a message will be retried up to 10 times before being dropped.
• --deliver-policy all – deliver all messages from the beginning.
• --replay policy – use the consumer’s replay policy (defaults to last).

4. Publish Test Messages

for i in {1..10}; do
  nats pub orders.new "{"\"id\":$i,"\"amount\":$((i*10))}" 
  sleep 0.1
done

This loop publishes ten JSON messages to orders.new which matches the stream’s subject pattern.

5. Pull Messages and Acknowledge

Run a simple consumer script that pulls 5 messages, prints them, and acknowledges.

#!/usr/bin/env bash
while true; do
  msgs=$(nats consumer pull ORDERS orders-consumer --max 5)
  if [[ -z $msgs ]]; then
    echo "No messages"; sleep 1; continue
  fi
  echo "$msgs" | while read -r msg; do
    echo "Received: $msg"
    # Simulate processing
    sleep 0.2
    # Acknowledge
    nats consumer ack ORDERS orders-consumer $msg
  done
done

Save as consumer.sh, make it executable, and run it in a separate terminal.

6. Simulate a Consumer Crash

While the consumer is running, press Ctrl‑C to terminate it. Immediately restart the script. Because the consumer had not acknowledged all messages, JetStream will redeliver them. Observe that the unacknowledged messages appear again and are processed.

7. Verify Stream and Consumer State

nats stream info ORDERS
nats consumer info ORDERS orders-consumer

The stream output should show Msg Count equal to the number of published messages, and the consumer should report Delivered and Ack Pending values that match the unacknowledged messages.

Expected Checks

• Server logs contain no js errors after stream creation.
• Stream info shows Storage Type: File and the correct subject pattern.
• Consumer info shows Max Deliver: 10 and Ack Policy: explicit.
• After a crash, the consumer receives the same set of messages it had not acknowledged.

Recovery Options

• If the stream becomes full, increase --max-msgs or change --retention to interest or workqueue to free space.
• To delete the stream, run nats stream delete ORDERS (requires admin rights).
• To reset a consumer’s state, delete it with nats consumer delete ORDERS orders-consumer and recreate.

Limitations & Practical Checks

• File storage consumes disk I/O; monitor iostat or similar tools to ensure the node can sustain the write rate.
• Memory usage grows with the number of in‑flight messages; use nats server --stats or Prometheus metrics to track js_msgs_in_flight.
• Setting --max-deliver too low may drop messages before your application can process them; adjust based on your retry strategy.
• Running the server with -DV in production is not recommended due to verbose logs; use -DV only during debugging.

Conclusion

With the steps above you have a durable JetStream stream and a pull consumer that survives crashes and supports replay. The configuration knobs allow you to fine‑tune retention, acknowledgment, and delivery semantics to match your application’s reliability requirements. Regularly verify server logs and metrics to pre‑empt disk or memory exhaustion in a high‑throughput environment.