Guaranteeing Delivery with NATS JetStream Durable Consumers

The Risk of the 'Fire-and-Forget' Pattern

In a standard NATS pub-sub model, messages are ephemeral. If a subscriber is offline when a message is published, that data is gone forever. For critical workflows—like processing a payment or updating an inventory count—this "fire-and-forget" behavior is a liability. You need a guarantee that every message is processed at least once, regardless of network partitions or service crashes.

The solution is JetStream, the persistence layer for NATS. By using Durable Consumers, you shift the responsibility of message tracking from the client to the NATS server, ensuring that the system remembers exactly where a specific worker left off.

How Durable Consumers Maintain State

A standard consumer is transient; it exists only as long as the client connection is active. A Durable Consumer, however, is a named entity registered on the NATS server. The server tracks the acknowledgment (ack) state for that specific name.

When a worker connects using a durable name, the server looks up the last acknowledged sequence number for that name and begins delivering messages from that point forward. If the worker crashes, the server keeps the messages in the stream. When a new worker joins with the same durable name, it picks up exactly where the previous instance stopped.

Implementation: The Order Processor Example

Consider a scenario where an orders.created subject receives incoming customer orders. We want a worker to process these orders reliably using the NATS Go client (assuming NATS Server v2.x+).

1. Define the Stream

First, the stream must be created to persist the messages. Run this from your terminal using the NATS CLI (requires --server and appropriate permissions):

nats stream add ORDERS --subjects "orders.*" --storage file --retention limits

2. The Worker Logic

The worker must explicitly acknowledge messages to signal successful processing. If Msg.Ack() is not called, the server will redeliver the message after the AckWait timeout expires.

// Example Go snippet for a durable consumer
js, _ := nc.JetStream()

// Create or bind to a durable consumer named "order-processor"
sub, _ := js.PullSubscribe("orders.created", "order-processor", nats.PullMaxWaiting(128))

for {
    msgs, _ := sub.Fetch(1)
    for _, msg := range msgs {
        // Process the order (e.g., save to DB)
        if err := processOrder(msg.Data); err == nil {
            // Mark as processed on the server
            msg.Ack()
        } else {
            // Log error; message will be redelivered based on AckWait
            log.Printf("processing failed: %v", err)
        }
    }
}

Operational Trade-offs and Constraints

Durability is not free. Moving from core NATS to JetStream introduces specific engineering overheads:

• Storage Pressure: Because messages are written to disk (or memory), you must define storage limits. If a stream reaches its max_bytes limit, NATS will either discard old messages or block new producers, depending on your discard policy.
• Latency: Writing to a persistent store is slower than the memory-only routing of core NATS. For ultra-low latency requirements, ensure your storage backend uses high-performance NVMe drives.
• Idempotency Requirement: At-least-once delivery means some messages will be delivered twice (e.g., if a worker processes a message but crashes before sending the Ack). Your business logic must be idempotent—processing the same order ID twice should not result in two charges to a customer.

Verifying Consumer Health

To ensure your durable consumers aren't falling behind, you can inspect the state via the CLI. Run the following command to check for "pending" messages (messages sent to the consumer but not yet acknowledged):

nats consumer info ORDERS order-processor

Look for the Num Pending metric. If this number grows indefinitely, your consumer is slower than your producer, and you may need to scale your worker pool or optimize the processing logic.

Rollback Procedure

If a durable consumer configuration causes an infinite loop of redeliveries (a "poison pill" message), you can reset the consumer state to the end of the stream to skip problematic messages:

nats consumer edit ORDERS order-processor --deliver last