Managing Consumer Offsets with NATS JetStream Pull Consumers

Problem: Consumers that crash lose their place in the message stream

When a NATS subscriber goes down, the application must decide whether to replay messages from the beginning (risking duplicate work) or start fresh (risking missed data). Without server‑side state, each restart forces the developer to rebuild offset tracking in the application layer, which is error‑prone and adds operational overhead.

Takeaway: JetStream pull consumers store offsets on the server, letting you resume exactly where you left off

By enabling JetStream and using the Pull consumer model, NATS tracks the last acknowledged message for each consumer. A crash does not lose progress; a restart simply requests the next batch of unacknowledged messages.

Setting up a stream and a pull consumer

First, start a NATS server with JetStream enabled:

# Run on any host with access to the nats binary
nats-server -js

Create a stream that retains messages for 24 hours or until 1 GB is stored:

# Add a stream named ORDERS
nats stream add ORDERS --subjects 'orders.>' --max-age 24h --max-bytes 1GB

Now add a pull consumer that acknowledges each message explicitly:

nats consumer add ORDERS --pull --ack explicit --name orders-worker

The consumer is created with an initial offset of zero. No messages have been acknowledged yet.

Publishing and consuming messages

Publish a few test messages:

for i in {1..5}; do nats pub orders.new "order-$i"; done

Fetch a batch of two messages using the pull consumer:

# Pull up to 2 messages, wait 2 seconds for a response
nats consumer pull orders-worker --batch 2 --timeout 2s

You should receive two messages, e.g.:

[#1] orders.new: order-1
[#2] orders.new: order-2

Acknowledge them explicitly:

nats consumer ack orders-worker 1 2

Check the consumer state to confirm the acknowledged count:

nats consumer info ORDERS orders-worker

The output shows NumAcked: 2 and NumPending: 0.

Simulating a crash and verifying resume

To mimic a crash, stop the consumer process before acknowledging the remaining messages. Then fetch another batch:

# Pull the next two messages (order-3, order-4) without acking them
nats consumer pull orders-worker --batch 2 --timeout 2s

Now terminate the consumer (Ctrl‑C). Restart it and pull again:

nats consumer pull orders-worker --batch 2 --timeout 2s

You will receive order-3 and order-4 again, because they were never acknowledged. After acknowledging those, the next pull yields order-5. The server‑side offset ensured you never skipped or duplicated a message that had already been processed.

Trade‑off: MaxAckPending and replication latency

JetStream lets you tune MaxAckPending, the maximum number of messages a consumer can have outstanding before the server stops sending more. Setting this too low can cause under‑utilization; setting it too high can lead to redelivery loops if the consumer fails to acknowledge before the server’s MaxAckPending window expires and the messages are considered unacknowledged.

In a clustered JetStream deployment, increasing the replication factor improves fault tolerance but adds latency because each write must achieve a Raft quorum. For workloads that require sub‑millisecond publish latency, a replication factor of 1 (single‑node) may be acceptable if the underlying storage is backed by reliable hardware.

Practical verification steps

1. Run nats server info to confirm JetStream is active (JetStream: true).
2. After creating the stream, run nats stream info ORDERS and verify Storage: File (or Memory) and the configured retention limits.
3. After acknowledging messages, check nats consumer info ORDERS orders-worker for rising NumAcked and stable NumPending.
4. To test the crash‑recovery scenario, follow the steps in the “Simulating a crash” section and observe that the consumer resumes at the last unacknowledged message.

Actionable closing

If your application needs reliable, exactly‑once processing without building custom offset stores, adopt NATS JetStream pull consumers:

• Enable JetStream with -js.
• Define a stream with appropriate retention (limits or interest).
• Create a pull consumer with --ack explicit.
• Monitor NumAcked and NumPending via the CLI or API to detect stuck consumers.
• Tune MaxAckPending based on your consumer’s processing time and avoid values that cause excessive redelivery.

By letting the server manage offsets, you reduce code complexity and gain a clear, observable point to resume work after any failure.