Durable Consumers in NATS JetStream: State, Replay, and Practical Pitfalls

Problem: Restarts Break Event‑Driven Workflows

In a microservice that consumes a NATS JetStream stream, a crash or upgrade can leave the consumer mid‑stream. If the consumer had no record of where it left off, it would either miss messages or reprocess everything, breaking idempotency guarantees.

Thesis: Durable Consumers Preserve State and Enable Replay

JetStream’s durable consumer feature records the last acknowledged sequence on the server. When a client reconnects, it can resume from that point or replay the entire retained history. This makes stateful services, audit logs, and event‑sourced systems robust.

1. What Makes a Consumer “Durable”?

• AckPolicy – None, Explicit, All. Explicit gives the client control but requires careful handling to avoid duplicates.
• DeliverPolicy – New (only new messages), All (retain history). All is required for replay.
• ReplayPolicy – Instant (deliver on connect) or Replay (client requests).
• MaxDeliver – limits redelivery attempts.
• StartSequence / StartTime – where the consumer should begin if no state exists.

These options are set when the consumer is created, either via the API or the nats consumer create CLI.

2. Configuring a Durable Consumer – CLI & API Example

Below is a minimal, reproducible example that demonstrates a durable consumer that can resume and replay.

# Step 1: Create a stream with a long retention window
nats stream add --name ORDERS \
  --subjects orders.* \
  --storage file \
  --retention limits \
  --max_age 168h   # 7 days

# Step 2: Publish some test messages
for i in {1..10}; do
  nats pub orders.created "{\"order_id\":$i}"
done

# Step 3: Create a durable consumer that will replay all history
nats consumer create ORDERS \
  --name order-processor \
  --durable \
  --ack_policy explicit \
  --deliver_policy all \
  --replay_policy instant \
  --max_deliver 5

# Step 4: Consume messages (example using nats.go)
package main

import (
    "context"
    "fmt"
    "github.com/nats-io/nats.go"
)

func main() {
    nc, _ := nats.Connect(nats.DefaultURL)
    js, _ := nc.JetStream()
    sub, _ := js.PullSubscribe("orders.*", "order-processor")
    for {
        msgs, _ := sub.Fetch(1, nats.MaxWait(1*time.Second))
        for _, m := range msgs {
            fmt.Printf("Received: %s\n", string(m.Data))
            m.Ack()
        }
    }
}

After running the consumer once, stop it, then restart. The consumer will resume from the last acknowledged sequence, not from the beginning.

3. Replay vs. Resume: How Deliver & Replay Policies Work

When DeliverPolicy=All and ReplayPolicy=Instant, the server sends every retained message immediately after the consumer connects. If you want to replay only on‑demand, set ReplayPolicy=Replay and call sub.Fetch as needed.

Example:

# Replay the entire history after a restart
nats consumer info ORDERS order-processor
# Output will show Delivered=10, Pending=0

# To trigger a replay manually
nats consumer replay ORDERS order-processor

Check the consumer’s state with nats consumer info to verify the Delivered sequence matches the last processed message.

4. Trade‑Offs & Common Pitfalls

• Memory & Disk Usage: A durable consumer that consumes all retained messages can exhaust the stream’s retention limits. Use max_age or max_msgs to bound the history.
• Duplicate Delivery: With AckPolicy=Explicit, if the client crashes before acknowledging, the server will redeliver the message. Implement idempotent processing or a deduplication window.
• Ack Policy Selection: None is fast but risks message loss if the client dies before processing. All ensures all messages are acknowledged but can be slower.
• Consumer Replay Limits: Replay can be expensive for large streams; consider max_deliver to prevent endless loops.

Actionable Take‑Away

1. Always create durable consumers with a clear DeliverPolicy and ReplayPolicy that match your use case.
2. Monitor Delivered and Pending via nats consumer info to spot state drift.
3. Set stream retention (max_age, max_msgs) to avoid runaway memory consumption.
4. Implement idempotent handling or a deduplication cache when using AckPolicy=Explicit.
5. Test restarts locally: publish > create consumer > stop > restart > verify Delivered sequence.

Durable consumers are a powerful tool in NATS JetStream, but they require thoughtful configuration to balance reliability, resource usage, and processing semantics.