Diagnosing Unexpected Message Redelivery in NATS JetStream Consumers

Problem Statement

When a NATS JetStream consumer processes a message, you expect it to be acknowledged once and never delivered again. In practice, you may see the same message appear multiple times, or a consumer that appears to stall because it never receives new messages. The root of these symptoms is usually a mismatch between the consumer’s acknowledgment strategy and the way the application handles acks.

Common Root Causes

• Ack not sent before ack_wait expires – the server assumes the message is unacknowledged and redelivers it.
• Auto‑ack wrappers or early context cancellation – the application believes it has acknowledged the message but the Msg.Ack() call never reaches the server.
• MaxAckPending reached – the consumer blocks because it has exceeded the number of unacknowledged inflight messages.
• Negative acknowledgments (Nak) with too short a delay – the consumer explicitly tells the server to redeliver immediately.
• Server restarts or leader changes – acks that were not persisted in a durable stream are lost, causing redelivery.
• Connection drops before ack propagation – the server treats the inflight message as pending and redelivers on reconnect.
• Out‑of‑order delivery after redelivery – multiple pull batches are in flight and acknowledgments are out of sequence.

Diagnostic Checklist

Condition	Diagnostic Check
Duplicate processing of a message	Run nats consumer info STREAM CONSUMER and look at NumRedelivered and NumAckPending. Verify logs for Msg.Ack() calls.
Pull consumer stalls, no messages arrive	Check NumAckPending == MaxAckPending in consumer info. Inspect server logs for “consumer blocked on ack pending”.
Push consumer receives same message rapidly	Search client code for Nak() or Msg.Nak(). Verify ack_wait and max_deliver settings.
Message redelivered after server restart	Confirm stream config: Storage=File and Replicas>=3. Run nats stream report STREAM to view AckFloor.
High NumRedelivered but no Nak in logs	Enable server debug logs (-DV) and filter for “ack” and “nak”. Correlate client disconnect timestamps.
Out‑of‑order delivery after redelivery	Check consumer DeliverPolicy and MaxBatch. Ensure only one batch is in flight.

Fixes & Best Practices

1. Set explicit ack_wait – align it with your maximum processing time plus a safety margin. Example:

   consumer:
     name: "orders-processor"
     ack_policy: Explicit
     ack_wait: 45s
     max_ack_pending: 2000
   

2. Instrument the ack call – log the timestamp before and after Msg.Ack() to ensure it reaches the server. In Go:

   log.Printf("Acking msg %s", msg.Subject)
   if err := msg.Ack(); err != nil {
       log.Fatalf("Ack failed: %v", err)
   }
   

3. Handle context cancellation – if you use a deadline, call Msg.Ack() before the context expires. Use a separate goroutine to guarantee the ack is sent.
4. Adjust max_ack_pending – if you process messages slowly, increase this value or enable max_deliver to cap redelivery attempts. Monitor the NumAckPending metric to avoid silent backpressure.
5. Use durable streams – set Storage=File and Replicas=3 for critical data. This ensures acks survive server restarts.
6. Avoid unnecessary Nak – only use Nak() when you truly need immediate redelivery. Otherwise let the server handle it via ack_wait.
7. Enable server debug logs – run the server with -DV and filter for “ack”, “nak”, and “redeliver” to trace the ack flow when troubleshooting.
8. Use a single outstanding batch for pull consumers – set max_batch: 1 or process batches synchronously to maintain order.

Escalation Criteria

• If NumRedelivered remains high after adjusting ack_wait and max_ack_pending, consider that the consumer may be dropping acks due to network issues. Verify TLS/WS health and client keepalive settings.
• Persistent redelivery after server restarts indicates a storage or replication misconfiguration. Engage the NATS Ops team to review the cluster’s Raft logs.
• If messages are still delivered out‑of‑order after ensuring single batch processing, investigate the application’s message ordering logic or the stream’s DeliverPolicy.

Practical Verification

After applying a fix, run the following to confirm the issue is resolved:

# 1. Check consumer state
nats consumer info STREAM CONSUMER
# 2. Simulate a long processing time > ack_wait
# 3. Observe NumRedelivered does not increase
# 4. Verify logs show Msg.Ack() executed before timeout

These steps provide a repeatable way to validate that acknowledgments are handled correctly and that the consumer no longer experiences unwanted redeliveries.

Conclusion

Unexpected message redelivery in NATS JetStream usually boils down to acknowledgment timing and consumer configuration. By systematically inspecting consumer info, server logs, and application code, you can pinpoint whether the root cause is a timing mismatch, a misconfigured consumer, or a storage durability issue. Adjusting ack_wait, max_ack_pending, and ensuring acks actually reach the server will restore reliable processing and prevent duplicate work.