Guaranteeing At‑Least‑Once Delivery in NATS JetStream with Durable Subscriptions

Problem: How can a NATS client keep receiving messages even after a crash?

In many distributed systems, a consumer may restart or lose network connectivity. If the message broker simply pushes a message once, a restart could mean that message is lost forever. NATS JetStream solves this with durable subscriptions that persist the consumer state on disk, enabling the broker to replay missed messages when the client reconnects.

Thesis: Durable consumers with the ack_policy=explicit setting give you a true at‑least‑once guarantee.

When a consumer acknowledges a message, JetStream removes it from the replay queue. If the consumer fails to acknowledge before the ack_wait period, JetStream will redeliver it. Because the consumer name is persisted, the same logical consumer will pick up where it left off after a restart, regardless of the client ID.

Key Concepts

• Stream – a log of messages that JetStream stores on disk.
• Consumer – a subscription that pulls messages from a stream. A durable consumer keeps its state on disk.
• Ack policy – controls when a consumer must acknowledge. explicit means the consumer must call Msg.Ack() to confirm receipt.
• At‑least‑once – the broker guarantees that each message will be delivered at least once, but duplicates can occur if a consumer dies before acknowledging.

Concrete Example: Test the Workflow End‑to‑End

We’ll use the nats CLI to demonstrate the flow. Replace <STREAM> and <CONSUMER> with your own identifiers.

# 1. Create a stream that holds 100 messages for 1 day
nats stream add <STREAM> --subjects "demo.>" --retention limits --max-msgs 100 --max-age 1d

# 2. Publish a test message
nats pub demo.test "Hello, JetStream!"

# 3. Subscribe with a durable consumer, at‑least‑once ack policy
#    The consumer name <CONSUMER> is stored on disk.
#    The client ID can be anything; the durable name is what matters.
#    Run this in a terminal window and keep it open.
#    (Use a unique client ID to avoid conflicts.)

nats sub durable <CONSUMER> --stream <STREAM> --ack --ack-policy explicit

# 4. Observe the message appear in the terminal.
#    Press Ctrl+C to exit the subscription.

# 5. Re‑run the subscription to see replay.
#    Since the consumer was durable and the message was not acknowledged,
#    JetStream will redeliver the message.

nats sub durable <CONSUMER> --stream <STREAM> --ack --ack-policy explicit

After step 5 you should see the same "Hello, JetStream!" message again. That demonstrates the replay capability.

Trade‑offs and Limitations

• Disk Footprint – Durable consumers retain the entire message history that has not been acknowledged. In high‑throughput scenarios, this can grow quickly.
• Replaying Large Histories – JetStream will replay all unacknowledged messages on startup. If a consumer has been down for a long time, the initial backlog may delay the first useful message.
• Consumer Name Conflicts – Using the same durable name across multiple clients (different client IDs) will cause a conflict. JetStream will reject the second subscriber with a ConsumerNameInUse error. Keep durable names unique per logical consumer.
• Duplicate Messages – At‑least‑once guarantees duplicates are possible if a consumer fails before acknowledging. Your application must be idempotent or handle duplicates.

Practical Checklist

1. Define a clear naming convention for streams and durable consumers.
2. Set ack_policy=explicit and ack_wait to a reasonable value (e.g., 30s).
3. Monitor the Delivered and Pending counters via nats consumer info to detect backlog growth.
4. Periodically prune old messages with max_age or max_msgs stream limits.
5. Implement idempotent message processing to safely handle duplicates.

Actionable Closing

Start by creating a small stream and a durable consumer in your dev environment. Use the CLI example above to confirm replay works. Once you’re comfortable, scale the stream size and observe how the consumer backlog behaves. Remember to keep your consumer names unique and your application tolerant of duplicates. With these steps, you’ll have a production‑ready at‑least‑once delivery pipeline on NATS JetStream.