Architecting Durable Message Queues with NATS JetStream

The Problem: Moving Beyond Fire-and-Forget

Standard NATS messaging is 'at-most-once,' meaning if a subscriber is offline when a message is published, that data is lost. For critical business processes—such as payment processing or order fulfillment—you need 'at-least-once' delivery. This requires a system that persists messages to disk and tracks which specific consumers have acknowledged the data.

The Smallest Suitable Design

To implement a durable queue, you do not need a complex cluster immediately. The minimum viable architecture consists of a NATS server started with the -js flag and a defined Stream.

Stream Configuration

A Stream captures messages on specific subjects and stores them. For guaranteed delivery, use these settings:

• Storage: File. Memory storage is faster but volatile; a full cluster restart will wipe your queue.
• Retention Policy: Limits. This keeps messages until a size or age limit is reached, regardless of whether they were consumed. For a strict queue where messages are deleted immediately after processing, use WorkQueue.
• Replicas: 1 (for single-node) or 3 (for high availability).

Consumer Strategy

To ensure no message is missed, use a Pull Consumer. Unlike Push consumers, which can overwhelm a client, Pull consumers request a specific batch of messages when they have the capacity to process them.

Trust and Data Boundaries

NATS manages security through a decentralized model using Accounts. In a production environment, you should isolate your streams by account to prevent unauthorized subjects from being read or written.

• Account Isolation: Each account has its own unique set of streams. A client in Account A cannot see the JetStream data of Account B.
• Authorization: Use NATS JWTs (JSON Web Tokens) to restrict a user's permissions. For example, a producer should have pub permissions for the subject, while a worker should only have sub permissions and the ability to manage its specific consumer state.

Operational Checks and Verification

Monitoring a durable queue shifts from watching 'traffic' to watching 'lag.' You must track the gap between the last message written to the stream and the last message acknowledged by the consumer.

Diagnostic Commands

Run these commands using the NATS CLI (installed on a management machine with admin permissions) to verify the state of your queue:

# Check the overall health and size of the stream
nats stream info MY_STREAM

# Identify lagging consumers and pending messages
nats consumer info MY_STREAM MY_CONSUMER

Verification Workflow

1. Start the NATS server with nats-server -js.
2. Create a stream: nats stream add MY_STREAM --subjects 'orders.*' --storage file.
3. Publish a message: nats pub orders.new 'Order Data'.
4. Disconnect your consumer application.
5. Reconnect the consumer and verify that it receives the message published during its downtime.

Failure Modes and Design Constraints

The Slow Consumer Problem

In a WorkQueue configuration, if a consumer pulls a message but fails to send an ACK (acknowledgment) within the ack_wait window, JetStream will redeliver that message. If the consumer is consistently too slow, you will see a 'redelivery loop,' where the same failing message blocks the queue or consumes resources.

Disk I/O Bottlenecks

Because File storage is required for durability, the primary bottleneck is disk IOPS. If you observe high latency in publishing, check your disk wait times. Moving to NVMe storage is the most effective hardware remedy for JetStream throughput issues.

When to Change the Design

The single-cluster, File-storage design works until you hit these three triggers:

Trigger	Required Change
Regional Outage Risk	Implement JetStream Mirroring to replicate a stream from one geographic region to another.
Extreme Throughput	Split a single large stream into multiple smaller streams partitioned by subject to distribute disk I/O.
Strict Ordering Requirements	Switch from a distributed consumer group to a single-active consumer to prevent out-of-order processing.

Rollback: To revert a stream configuration, you must delete and recreate the stream via nats stream rm MY_STREAM, as most storage and retention settings cannot be changed on the fly without a recreation.