Implementing Exactly-Once Processing with NATS JetStream Acknowledgment Policies

The Challenge of Distributed Message Delivery

In distributed systems, the primary risk during message processing is the "uncertainty gap": the period between a consumer finishing a task and the server recording that the task is complete. If a consumer crashes or the network fails after the work is done but before the acknowledgment (ack) is sent, the server will redeliver the message, leading to duplicate processing.

To achieve exactly-once semantics, you must combine NATS JetStream's explicit acknowledgment policy with application-level idempotency. This ensures that while a message might be delivered more than once (at-least-once delivery), it is only processed effectively once.

Prerequisites

• NATS Server v2.10 or later with JetStream enabled.
• The NATS CLI installed and configured.
• A defined stream to hold the messages. For this guide, we assume a stream named ORDERS covering subjects orders.>.

Configuring the Consumer for Reliability

To prevent automatic acknowledgments and control exactly when a message is marked as complete, you must create a durable consumer with an explicit acknowledgment policy. Run the following command on your terminal with appropriate administrative permissions:

nats consumer add ORDERS durable_orders --ack explicit --max-deliver 3 --ack-wait 30s

Key Configuration Parameters:

• --ack explicit: Disables automatic acks. The server will not consider a message processed until the client explicitly sends an ack().
• --max-deliver 3: Limits how many times NATS will attempt to deliver a single message. This prevents "poison pills" (malformed messages that crash consumers) from looping indefinitely.
• --ack-wait 30s: The timeout period. If the server doesn't receive an ack within 30 seconds, it assumes the consumer failed and schedules redelivery.

The Processing Workflow

When implementing the consumer logic in your application, follow this sequence to maintain processing integrity:

1. Pull/Subscribe: Subscribe to the subject using the durable consumer name and manual acknowledgment mode. nats sub orders.new --consumer durable_orders --manual-ack
2. Idempotency Check: Before processing, check a database or cache (e.g., Redis) using a unique message ID to see if this specific message has already been processed.
3. Execute Logic: Perform the business operation (e.g., updating an order status).
4. Acknowledge: Call msg.ack() only after the business operation and the idempotency record are committed.

Handling Long-Running Tasks

If a specific message requires more time than the --ack-wait window (e.g., a complex API call), do not simply increase the global timeout for all messages. Instead, use the in_progress() method. This tells the NATS server that the consumer is still working and resets the timer for that specific message, preventing premature redelivery.

Diagnostic Checks and Verification

To verify that your acknowledgment policy is functioning as expected, use the NATS CLI to inspect the consumer state:

nats consumer info ORDERS durable_orders

What to look for in the output:

Metric	Expected Behavior
Ack Policy	Must show Explicit.
Num Pending	Should decrease immediately after a successful ack().
Num Redelivered	Increments if a consumer crashes before acknowledging.

Recovery and Failure Handling

Scenario: Consumer Crash

If a consumer process is killed after processing the data but before the ack(), the message will remain in the stream. Once the ack-wait period expires, NATS will redeliver the message to the next available consumer. Because you implemented an idempotency check in Step 2 of the workflow, the second consumer will see the message was already processed and simply call ack() without duplicating the work.

Scenario: Poison Pills

If a message consistently causes a crash, it will hit the --max-deliver 3 limit. To handle this, configure a Dead Letter Queue (DLQ) on your stream using the --dead-letter flag. Once the max delivery count is reached, NATS will move the message to the DLQ stream for manual inspection rather than discarding it.

Engineering Trade-offs: Explicit vs. All

While explicit is the gold standard for exactly-once processing, it introduces more network chatter. For high-throughput systems where slight duplication is acceptable or handled by a downstream database constraint, you may consider Ack All. This acknowledges all messages up to a certain sequence number in a single call, reducing overhead but increasing the risk of redelivering a batch of messages if the consumer fails mid-batch.

Rollback Procedure

If the explicit acknowledgment policy is causing bottlenecks or unexpected redeliveries, you can delete and recreate the consumer with a different policy:

nats consumer rm ORDERS durable_orders

Note: Deleting the consumer removes the tracking of which messages were acknowledged. Upon recreation, the consumer will start from the stream's configured start sequence (e.g., from the beginning or the latest message).