Diagnosing Pub/Sub Acknowledgment Timeout and Unintended Redelivery

Problem and Takeaway

Messages that take longer than the subscription’s acknowledgment deadline are automatically redelivered, causing duplicate processing. The fix is usually a combination of raising the deadline, extending it from the subscriber while processing, and verifying that the ack call actually reaches Pub/Sub. Start by confirming the current deadline and the handler’s runtime; then apply the smallest change that eliminates redelivery.

Recognizable Condition

• Same message appears multiple times in your logs with identical messageId.
• Cloud Monitoring shows a rising pubsub.googleapis.com/subscription/redelivery_count metric.
• Subscriber logs show the handler finishing after the default 10 s deadline (or your custom value).

Cause / Diagnostic Table

Symptom	Likely Cause	Quick Check
RedeliveryCount spikes	Ack deadline shorter than handler latency	Compare ackDeadlineSeconds on the subscription with measured handler duration.
Handler logs stop before ack	Missing or misplaced ack() call	Search subscriber code for message.ack() (or acknowledge()) and ensure it runs on every path.
Intermittent redelivery under load	Network latency or Pub/Sub service hiccup delays ack	Correlate pubsub.googleapis.com/subscription/ack_latency spikes with redelivery events.
Dead‑letter queue fills	Repeated failures exceed max delivery attempts	Check subscription’s deadLetterPolicy.maxDeliveryAttempts and see if attempts match redelivery count.

Ordered Troubleshooting Steps

1. Inspect the subscription’s ack deadline

   Run the following in a shell with the gcloud CLI (requires pubsub.subscriptions.get permission):

   gcloud pubsub subscriptions describe SUBSCRIPTION_ID \
       --project=PROJECT_ID \
       --format='value(ackDeadlineSeconds)'

   If the value is ≤ 10 s and your handler regularly exceeds that, the deadline is the primary suspect.

2. Verify acknowledgment logic in the subscriber

   Add temporary log lines immediately before and after the ack call:

   logger.info("About to ack message %s", message.message_id)
   message.ack()
   logger.info("Ack sent for %s", message.message_id)

   Deploy and watch logs; missing the second line means the ack never fires.

3. Measure actual handler latency

   Instrument the handler with a timer (e.g., time.perf_counter() in Python) and emit the duration as a custom metric or structured log. Compare the 95th‑percentile latency to the deadline.

4. Check network / service latency

   In Cloud Monitoring, chart pubsub.googleapis.com/subscription/ack_latency for the subscription. Sustained latency > 1 s often correlates with premature redelivery.

5. Extend the deadline from the subscriber (if handler is long‑running)

   Use the client library’s modify_ack_deadline (or ModifyAckDeadline API) in a background thread or at regular intervals while processing:

   # Python example
   from google.cloud import pubsub_v1
   subscriber = pubsub_v1.SubscriberClient()
   subscription_path = subscriber.subscription_path(PROJECT_ID, SUBSCRIPTION_ID)
   
   def extend_deadline(message, extra_seconds=30):
       subscriber.modify_ack_deadline(
           request={
               "subscription": subscription_path,
               "ack_ids": [message.ack_id],
               "ack_deadline_seconds": extra_seconds,
           }
       )
   
   # Call extend_deadline every 20 s while work proceeds
   

   Run this only after confirming the handler is idempotent; otherwise you risk processing the same message twice if the extension fails.

6. Raise the subscription deadline (if handler consistently needs more time)

   Update the subscription (requires pubsub.subscriptions.update):

   gcloud pubsub subscriptions update SUBSCRIPTION_ID \
       --project=PROJECT_ID \
       --ack-deadline=60

   Start with a modest increase (e.g., 30 s) and observe RedeliveryCount. Avoid values > 600 s unless you have a dead‑letter topic to catch permanently stuck messages.

Fixes Tied to Findings

Finding	Fix	Risk / Limitation
Deadline < handler latency	Increase ackDeadlineSeconds or extend dynamically	Very high deadlines delay detection of genuine failures and grow backlog.
Missing ack call	Add message.ack() on all success/exception paths	None if idempotent; otherwise ensure exactly‑once semantics.
Network ack latency spikes	Enable client‑side flow control (max_messages) and retry ack with exponential backoff	May increase memory usage; test under load.
Repeated redelivery despite fixes	Configure a dead‑letter topic and alert on dead_letter_count	Dead‑letter messages still need manual inspection.

Verification Checklist

• After each change, wait one full processing cycle (typically 2–3× the deadline) and confirm RedeliveryCount drops to zero.
• Correlate AckLatency with the new deadline; latency should stay well below the deadline.
• Run a short load test (e.g., publish 1 k messages) and verify each message is logged exactly once.

Escalation Criteria

Open a support case when all of the following are true:

• RedeliveryCount remains > 0 after deadline increase, ack‑extension, and code audit.
• AckLatency shows no abnormal spikes (ruling out network).
• Dead‑letter topic receives messages, indicating the service itself cannot deliver acks.

Provide the subscription name, project ID, and the Monitoring charts for RedeliveryCount and AckLatency to accelerate triage.

Limitations & Practical Result Check

Raising the deadline masks slow handlers but does not fix them; long‑running work should eventually be moved to a background worker (e.g., Cloud Run, Cloud Functions) with a short Pub/Sub front‑end. The practical way to confirm the fix is to watch the pubsub.googleapis.com/subscription/redelivery_count metric in Cloud Monitoring for at least 15 minutes after the change—if it stays at zero, the timeout issue is resolved.