Reliable NATS JetStream Processing With Durable Consumers and Explicit Acks

Messages disappear when a consumer crashes mid-batch. With plain NATS subscriptions, an in-flight message is lost if the subscriber dies before processing finishes, and a restart starts from the next message. The practical fix is JetStream with a durable consumer and explicit acknowledgments: the server tracks delivery state on disk, so a restarted worker resumes where it left off and only messages that were not acknowledged are redelivered.

What breaks without durability

JetStream is the persistence layer for NATS. A stream stores messages to file or memory and retains them according to retention policy. A consumer is a view into a stream with its own cursor.

An ephemeral, auto-acknowledged consumer has no cursor. If the client disconnects, the server forgets which messages were in flight. That is fine for fan-out telemetry, but it is unsafe for work queues. Durable consumers solve this by naming the consumer and persisting its state.

How durable consumers checkpoint state

A durable consumer is identified by a durable name. The server stores the last acknowledged sequence and the set of pending messages for that name. With acknowledgment policy set to explicit, the server will not advance the consumer until the client sends an Ack.

Two knobs matter for reliability engineering. Max Ack Pending limits how many messages the server will deliver without an Ack, providing back-pressure. A filter subject narrows the stream to the messages a worker cares about.

Because JetStream can redeliver after a crash, handlers must be idempotent. A non-idempotent side effect such as a double charge can occur if the same message is processed twice. Design for at-least-once delivery, not exactly-once.

Storage choice is a trade-off

JetStream supports file-based storage for durability and memory-based storage for speed. File storage writes stream data and consumer state to disk under the server's -store_dir. That adds a few milliseconds of latency and consumes disk space, but survives restarts.

Memory storage reduces latency and is useful for low-latency, replay-free workloads where loss on server restart is acceptable. For reliable processing, file storage is the usual choice.

Verify the store directory has free space and the nats-server process has write permission before creating streams. Consumer creation will fail with a storage error if the directory is unwritable.

Worked example with the NATS CLI

Run the following on a host with nats-server and the nats CLI installed. The server needs write access to the store directory.

nats-server -js -store_dir ./jsstore

Confirm JetStream is active with

nats server info

Look for JetStream enabled in the output.

Create a file-backed stream for hello subjects.

nats stream add hello.stream --subjects hello.*> --storage file

Publish a small test set.

for i in {1..10}; do nats pub hello.world "msg-$i"; done

Add a durable pull consumer with explicit acks.

nats consumer add hello.stream --durable worker1 --ack explicit --filter hello.world

Pull messages with a subscriber that acknowledges manually. In practice this is done in an application SDK with Ack() called after successful processing. The CLI pull mode can be used for inspection.

nats sub hello.world --stream hello.stream --consumer worker1 --pull

Kill the subscriber after processing a subset, restart it with the same durable name, and inspect consumer state.

nats consumer info hello.stream worker1

The consumer info shows NumPending and NumAckPending. After restart, unacknowledged messages remain pending and can be redelivered and acknowledged. Once all messages are acknowledged, NumAckPending drops to zero.

Limitations to plan for

Durable consumers add disk I/O for state checkpointing. That is a cost for persistence. For sub-millisecond, loss-tolerant workloads, a memory stream with non-durable consumers may be more appropriate.

Storage grows with stream retention. Monitor disk usage for the store directory and set appropriate max bytes or max messages on the stream.

Always make processing idempotent and keep max ack pending small enough to bound rework after a crash.

Start with file storage, explicit acks, a named durable consumer, and idempotent handlers. Check consumer info after restarts to confirm pending messages are redelivered and then acknowledged.