Implementing At-Least-Once Delivery with NATS JetStream Pull Consumers

The Challenge of Message Loss in Core NATS

Core NATS is a fire-and-forget messaging system. If a subscriber is offline or crashes during processing, the message is lost. For critical workflows—such as payment processing or order fulfillment—you need a guarantee that every message is processed at least once, regardless of network partitions or consumer failures.

The solution is JetStream, the persistence layer for NATS. By moving from a subject-based pub/sub model to a stream-based model, you can store messages on disk and use Pull Consumers to control the flow of data and explicitly acknowledge successful processing.

Prerequisites

• NATS Server installed (v2.0+).
• The server must be started with JetStream enabled. Use the -js flag: nats-server -js.
• The NATS CLI installed for configuration and verification.

Step 1: Define the Persistent Stream

A Stream captures messages published to one or more subjects and stores them. The Retention Policy determines when messages are removed from the storage.

Run the following command on your local terminal to create a stream named ORDERS that listens to all subjects starting with orders.:

nats stream add ORDERS --subjects "orders.*" --storage file --retention limits --max-msgs=-1 --max-bytes=-1

Configuration Details:

• --storage file: Ensures messages survive server restarts by writing to disk.
• --retention limits: Keeps messages until they hit a size or count limit (set to -1 for unlimited in this example).
• --subjects "orders.*": A wildcard that captures orders.new, orders.cancelled, etc.

Step 2: Configure a Pull Consumer

Unlike Push Consumers, which flood the client with data, a Pull Consumer allows the client to request a specific batch size. This prevents the consumer from being overwhelmed and is the primary mechanism for ensuring at-least-once delivery.

Create a durable consumer named order-processor:

nats consumer add ORDERS order-processor --pull --ack explicit --deliver-all

Critical Decision: Ack Policy

Policy	Behavior	Use Case
explicit	Server waits for a specific ACK for every message.	High-reliability tasks where every single message must be verified.
all	Acknowledging message N acknowledges all messages before it.	Sequential processing where order is strict.
none	Server assumes delivery upon sending.	Low-latency telemetry where loss is acceptable.

Step 3: Processing and Acknowledgment

The application logic must follow a strict sequence: Fetch → Process → Acknowledge. If the application crashes after fetching but before acknowledging, JetStream will redeliver the message after the AckWait timeout expires.

To test this via the CLI, fetch a single message:

nats consumer next ORDERS order-processor

The CLI will display the message and its Sequence Number. To mark it as processed, you must send the acknowledgement. If you exit the process before the ACK is sent, the message remains in the stream and will be delivered to the next request.

Verification and Diagnostics

To verify that messages are being tracked and not lost, use the stream info command:

nats stream info ORDERS

Check the Messages count versus the First/Last Sequence. If the sequence increases but the consumer is not progressing, check for "unacknowledged" messages in the consumer details:

nats consumer info ORDERS order-processor

Limitations and Risks

• Disk Space: Using --max-bytes=-1 can lead to server crashes if the disk fills up. In production, always set a reasonable limit (e.g., --max-bytes 10GiB).
• Duplicate Processing: At-least-once delivery means a message might be delivered twice (e.g., if the consumer processes the data but crashes before sending the ACK). Your application logic must be idempotent—processing the same message twice should not change the result.
• Work Queue Retention: Avoid using --retention work_queue if you need to replay historical data, as this policy deletes messages as soon as they are acknowledged by any consumer.

Rollback Procedure

If the stream configuration is incorrect, you must delete the stream to reset the storage and subjects:

nats stream delete ORDERS

Warning: This permanently deletes all stored messages within that stream.