Diagnosing JetStream Pull Consumer Lag: Pending Messages That Never Drain

The condition: pending grows while publish rate stays flat

You have a JetStream pull consumer whose num_pending climbs steadily even though producers are publishing at a normal, steady rate. End-to-end latency rises, num_ack_pending sits near its cap, and you start seeing redelivered messages. This is consumer lag: the consumer's ack rate has fallen below the stream's ingest rate, and the backlog will keep growing until you close that gap or retention discards messages.

The useful takeaway: lag is a rate mismatch, and the fix depends on which side of the mismatch is broken — the consumer's processing speed, the consumer's configuration, or the server's ability to serve messages. Diagnose before tuning. This guide assumes NATS Server 2.9.x or later with JetStream enabled and a durable pull consumer; field names and defaults vary slightly across releases, so confirm against your deployed version.

Cause and diagnostic table

Ordered checks

1. Read the consumer state

Run nats consumer info MY_STREAM MY_CONSUMER from any host with NATS CLI credentials for the account that owns the stream. Note num_pending, num_ack_pending, num_redelivered, and the ack floor sequence. Take the same snapshot twice, 60 seconds apart. If the ack floor is advancing but pending still grows, the consumer works but is too slow. If the ack floor barely moves while num_ack_pending is pinned at the cap, the consumer is stalled or saturated with unacked work.

2. Compare rates, not snapshots

A single pending number tells you nothing. Measure publish rate from stream info (nats stream info MY_STREAM, message count delta over the window) and compare it to ack floor progression. If ack rate is, say, 70% of publish rate, no configuration trick will help — you need more consumer capacity or faster processing.

3. Measure processing time inside the application

Instrument your handler: time from message receipt to Ack(). Compute p50 and p99. If p99 exceeds the consumer's ack_wait, every slow message is redelivered while still being processed, which duplicates work and makes lag worse — a self-reinforcing failure.

4. Review consumer configuration

Check ack_wait, max_deliver, and max_ack_pending in the consumer config, plus your client's fetch batch size and expiry. A common misconfiguration: max_ack_pending of 1000 with only 5 worker goroutines — the consumer fetches far more than it can process, messages sit unacked until AckWait expires, and redeliveries explode.

5. Check the server side

If all consumers on the stream lag together, look at the server: JetStream file store disk latency, CPU, and memory. On a cluster, check that all replicas are current — a lagging replica can slow quorum writes. Server logs with JetStream warnings about storage or raft are an escalation signal, not a tuning target.

Fixes tied to findings

Processing-bound: add parallelism

Scale out subscriber instances (a queue-group-style pull consumer lets multiple clients share the work) and increase per-instance workers. Match fetch batch size to worker capacity — a reasonable starting point is batch ≈ workers × messages each worker handles within a fraction of AckWait. Verify by watching ack rate rise above publish rate and pending begin to drain.

Redelivery-driven: align AckWait with reality

Set ack_wait comfortably above p99 processing time — for example, if p99 is 12 seconds, use 30–60 seconds, not 15. Update it with nats consumer edit MY_STREAM MY_CONSUMER --ack-wait=45s (requires account permissions to modify the consumer). Trade-off: a longer AckWait delays redelivery of genuinely failed messages and holds more unacked state. Confirm success by watching num_redelivered stop growing while pending drains.

Over-fetching: cap MaxAckPending

Set max_ack_pending to roughly what your workers can clear within one AckWait window. This prevents the consumer from hoarding messages it cannot finish, which both causes redeliveries and starves other subscriber instances.

Storage-bound: fix the disk, not the consumer

If server-side persistence is the bottleneck, move the stream to faster storage (SSD-backed file store), reduce replication factor only if durability requirements allow, or shard traffic across multiple streams. Consumer tuning cannot outrun a slow disk.

Protect against recurrence

Add backpressure in the application: bound in-flight work, and shed load or pause fetches when downstream dependencies degrade. A circuit breaker on a slow database call prevents a dependency hiccup from becoming a redelivery storm.

Verifying the fix

After any change, re-run the two-snapshot comparison: pending should trend down, ack rate should meet or exceed publish rate, and redeliveries should drop toward zero. Watch end-to-end latency from publish to ack under steady load for at least one full traffic cycle. If pending drains only because traffic dipped, the fix is not proven.

When to escalate

Escalate beyond consumer tuning when: pending approaches the stream's retention limits (message loss is imminent under limits-policy retention); messages repeatedly hit max_deliver and are dropped or dead-lettered; cluster replicas show growing lag; or the server sustains high CPU/disk wait with JetStream warnings. At that point the problem is capacity or infrastructure, and it needs storage, cluster, or traffic-shaping work — not another config edit.