Diagnosing Stalled JetStream Pull Consumers: AckPending & Redelivery Issues

Recognizable Condition

A JetStream pull consumer that keeps showing an increasing AckPending count, redelivers the same messages after AckWait expires, and eventually stops delivering new messages is a classic sign that the consumer is stuck. Even though the consumer declaration remains active, the server will pause delivery when the pending window is full or when it suspects a slow consumer.

Root‑Cause Diagnostic Table

Cause	Symptoms	Key Metrics
AckWait < Processing Time	Messages redeliver before the client finishes processing.	High NumAckPending, frequent redeliver count in consumer info.
MaxAckPending Exceeded	Server pauses delivery; NumAckPending stays at the max value.	Consumer shows MaxAckPending reached; server logs contain "max ack pending" warnings.
Missing Ack/Nak/Term	Messages remain in AckPending indefinitely.	Zero Ack count in client logs; consumer info shows no change after delivery.
Connection or Network Stall	Consumer disconnects or stalls; server logs “slow consumer” or “connection lost”.	Consumer state changes to DISCONNECTED; redelivery counter increases.

Ordered Diagnostic Checks

1. Inspect Consumer State

   Run from a machine that can reach the NATS server:

   nats consumer info <stream> <consumer> --server <url>

   Verify NumAckPending, NumRedelivered, and MaxAckPending. A growing NumAckPending with a stable MaxAckPending indicates a backlog.

2. Compare Client Timing to AckWait

   Measure the average time your application takes to process a message (including parsing, DB writes, etc.). Compare this to the consumer’s AckWait value (default 30s). If processing takes longer, the message will be redelivered before you finish.

3. Confirm Acknowledgment Path

   Enable client‑side logging for Ack, Nak, or Term calls. Ensure every delivered message triggers exactly one of these calls and that the call succeeds (no connection errors).

4. Check MaxAckPending Threshold

   In the consumer info, MaxAckPending is the ceiling for pending messages. If your consumer’s prefetch batch size or the number of concurrent pulls is larger than this threshold, the server will halt delivery.

5. Review Server Logs

   Search the NATS server logs for warnings such as "max ack pending reached" or "slow consumer". These entries often include the consumer name and the reason for pausing delivery.

Fixes Tied to Findings

• Increase AckWait

  If processing time > AckWait, raise the value. Example:

  nats consumer update <stream> <consumer> --ackwait 120s --server <url>

• Adjust MaxAckPending or Prefetch

  Either increase MaxAckPending or reduce the pull batch size to keep the pending window below the threshold.

  nats consumer update <stream> <consumer> --maxackpending 2000 --server <url>

• Guarantee Acknowledgment

  Wrap message handling in a try/catch and send Ack in a finally block. For idempotent processing, use Nak on failure to trigger a retry.

• Restart or Recreate Consumer

  If the consumer has been stuck for a long time, delete and recreate it to clear stale state:

  nats consumer delete <stream> <consumer> --server <url>
  nats consumer create <stream> <consumer> --config <config_file> --server <url>

• Scale Streaming or Add Consumers

  For high‑throughput workloads, partition the stream or add additional pull consumers to distribute the load.

Escalation Criteria

Escalate to the NATS operations team if:

• AckPending continues to grow even after adjusting AckWait and MaxAckPending.
• Multiple consumers on the same stream exhibit the same stall pattern.
• Server logs show repeated slow‑consumer warnings after client‑side fixes.
• The issue persists across server restarts, indicating a configuration or version‑specific bug.

Verification Steps

1. After applying a change, run nats consumer info again and confirm NumAckPending decreases.
2. Check the /metrics endpoint for the stream: jetstream_consumer_ack_pending should drop.
3. Cross‑reference client logs: each delivered message should have a corresponding Ack entry with a matching message ID.

Limitations & Caveats

• Behavior of AckWait and MaxAckPending varies between NATS server releases (e.g., 2.x vs 3.x). Verify against your deployed version.
• Redelivery after AckWait expiry is expected; it does not indicate message loss.
• This guide applies only to JetStream pull consumers; core NATS publish/subscribe does not expose AckPending.

Practical Example

Suppose a consumer processes messages in ~45 s but the AckWait is set to 30 s. After a few pulls, NumAckPending climbs to 500, and the server logs “max ack pending reached”. The fix is straightforward:

# Increase AckWait to 60s
nats consumer update myStream myConsumer --ackwait 60s --server nats://localhost:4222

# Reduce prefetch to 200 to stay below MaxAckPending (default 2000)
nats consumer update myStream myConsumer --maxackpending 2000 --pullsize 200 --server nats://localhost:4222

After the update, monitor the consumer until NumAckPending stabilizes below 200 and redeliver count stops rising.