Pub/Sub Backlog Keeps Growing: Diagnosing Slow Subscribers, Redelivery Loops, and Poison Messages

The condition: oldest unacked age climbs while publish rate is flat

Your Pub/Sub subscription's backlog is growing and you need to know whether consumers are merely slow, stuck in a redelivery loop, or choking on a handful of poison messages. The recognizable condition is two metrics rising together while publish throughput stays flat: subscription/oldest_unacked_message_age climbs steadily and subscription/num_undelivered_messages grows, even though your consumer pods or instances look healthy. Acks are happening, just slower than deliveries — or the same messages are being delivered over and over without ever being acked.

This guide walks through a short cause table, ordered checks, fixes matched to each finding, and the point where you should escalate before messages silently expire.

Cause and diagnostic table

Ordered checks

Work through these in order; each one narrows the cause class before you change anything.

1. Compare publish vs ack rate. In Cloud Monitoring's Metrics Explorer, chart pubsub.googleapis.com/subscription/num_undelivered_messages and subscription/oldest_unacked_message_age for the affected subscription, alongside publish message count on the topic. If ack rate is close to publish rate but backlog still grows, suspect redelivery loops rather than raw slowness.
2. Confirm the subscription's actual configuration. Run gcloud pubsub subscriptions describe SUBSCRIPTION_NAME (any principal with pubsub.subscriptions.get, from Cloud Shell or your workstation). Check the ack deadline, dead-letter policy, retention duration, and any filter. Do not assume defaults — a shortened ack deadline or an unexpected filter changes the whole diagnosis.
3. Inspect subscriber flow control and CPU. Pub/Sub client libraries cap outstanding messages and bytes per consumer. If consumers show low CPU but the backlog grows, the cap is the bottleneck. If CPU is pinned, the consumers are the bottleneck.
4. Look for repeated deliveries of the same message IDs. Sample your consumer logs. The same message ID appearing across workers or across time means lease extensions are expiring before the handler finishes — the classic redelivery loop.
5. Check downstream dependency latency. If the handler writes to a database or calls an API, correlate handler duration with that dependency's latency. A slow downstream makes every message slower, which compounds into deadline expiry.

Fixes tied to findings

If flow control is the limit: raise the client library's max outstanding messages/bytes, or scale out consumer instances. Verify CPU headroom first — raising flow control on a saturated consumer just moves the bottleneck.

If handlers exceed the ack deadline: either make the handler faster (batch writes, remove synchronous calls) or increase the subscription's ack deadline. Client libraries with lease management extend deadlines automatically, but only up to a maximum extension duration — a handler slower than that ceiling will still loop. Prefer fixing handler latency over endlessly extending leases.

If poison messages dominate redeliveries: attach a dead-letter topic with a bounded delivery-attempt count. This converts an infinite retry loop into a finite one and gives you a diagnostic signal via subscription/dead_letter_message_count. Inspect the DLQ contents to find the failing payload pattern, fix the handler, then republish or seek to replay.

If a downstream dependency is saturated: fix the dependency or add backpressure (reduce flow control deliberately) so messages wait in Pub/Sub — where they are durable — instead of timing out mid-handler.

Escalation criteria

Escalate to the owning on-call team or Google Cloud Support when any of these hold:

• Backlog age approaches retention. Unacked messages are retained for a limited window (commonly 7 days by default, configurable up to 31 — confirm with the describe command above). If oldest_unacked_message_age approaches that window, you risk silent message loss. Treat this as an incident, not a tuning exercise.
• Ack rate looks healthy but backlog still grows. This pattern suggests configuration issues (filters, exactly-once or ordering interactions) that need deeper investigation.
• You are considering seek/replay. Seeking a subscription to a snapshot or timestamp replays or skips messages broadly. It is a deliberate recovery operation with data-loss or duplicate-delivery implications — not a diagnostic step. Get a second set of eyes first.

When escalating, include the subscription name, the time window, and screenshots of the two backlog metrics plus publish/ack rates.

Verifying recovery

After applying a fix, watch oldest_unacked_message_age fall toward zero while ack rate meets or exceeds publish rate for at least one full traffic cycle (for example, a full peak/off-peak rotation). A brief dip followed by renewed growth usually means the fix addressed throughput but a redelivery loop or poison message remains.

Cautions

• Metric names, default ack deadlines, and retention values are configuration- and version-sensitive. Confirm your subscription's actual settings before acting.
• Do not toggle exactly-once delivery or ordering keys mid-incident; both change throughput and redelivery semantics and can make the backlog worse.