Using Google Cloud Pub/Sub Dead‑Letter Topics to Capture Failed Messages

Why use a dead‑letter topic?

When a Pub/Sub subscriber cannot successfully process a message after a configurable number of delivery attempts, the message is discarded from the original subscription. A dead‑letter topic captures those messages so you can inspect, reprocess, or alert on them without losing data. This is the practical engineering decision for handling retries and failures in production pipelines.

Worked configuration

Attach a dead‑letter policy to a subscription at creation time (or recreate the subscription). The policy defines the dead‑letter topic, maximum delivery attempts, and an optional retry delay.

The following gcloud command creates a pull subscription orders-sub on topic orders with a dead‑letter topic orders-dlq, 5 maximum attempts, and a 2‑minute retry delay. Run it in a shell where the gcloud SDK is authenticated with a service account that has pubsub.subscriptions.create, pubsub.topics.create, and pubsub.topics.publish permissions on the project.

gcloud pubsub subscriptions create orders-sub \\\n    --topic=orders \\\n    --dead-letter-topic=orders-dlq \\\n    --max-delivery-attempts=5 \\\n    --retry-delay=2m

If orders-dlq does not exist, create it first:

gcloud pubsub topics create orders-dlq

After creation, publish a test message that will intentionally fail (e.g., a push endpoint that returns 5xx). The message will be retried up to five times, then appear in orders-dlq with its original payload and attributes intact.

Limits enforced by Pub/Sub

Parameter	Allowed range	Notes
maxDeliveryAttempts	1‑100	Per subscription; each subscription to the same topic tracks its own count.
retryDelay	0‑30 minutes	Zero means immediate retry; small values can cause tight loops.
Dead‑letter topics per subscription	1	Immutable after creation – change requires recreating the subscription.
Message retention	Default 7 days, configurable up to 31 days	Applies to both source and dead‑letter topics; set explicitly if longer storage is needed.

Common mistakes and how to avoid them

1. Missing dead‑letter topic – The gcloud call fails if the target topic does not exist. Create the topic first.
2. Too short retryDelay – A delay of 0 or a few seconds can cause rapid re‑delivery loops, increasing cost and log noise. Use at least 30 seconds for most workloads.
3. Assuming automatic acknowledgment – Messages moved to the dead‑letter topic are not acknowledged on the original subscription. They are simply removed from the original backlog; you must explicitly acknowledge or reprocess them from the dead‑letter subscription.
4. No monitoring on the dead‑letter topic – Pub/Sub does not surface a built‑in dashboard. Create a Cloud Monitoring alert on pubsub.googleapis.com/subscription/dead_letter_message_count (or the equivalent metric for the dead‑letter subscription) to detect spikes.
5. Deleting the dead‑letter topic – If the topic is removed before you drain its subscription, all captured messages are lost. Protect the topic with IAM policies and enable preventDeletion via resource hierarchy if required.

Verifying the setup

1. Publish a message that will fail delivery (e.g., to a push endpoint returning 500):

   gcloud pubsub topics publish orders --message='{\"orderId\": \"12345\"}'

2. Watch Cloud Logging for the subscription orders-sub. Filter on resource.type=\"pubsub_subscription\" AND jsonPayload.message.messageId=\"\". You should see up to maxDeliveryAttempts deliveryAttempt entries.
3. Pull from the dead‑letter subscription (create one if needed):

   gcloud pubsub subscriptions create orders-dlq-sub --topic=orders-dlq
    gcloud pubsub subscriptions pull orders-dlq-sub --auto-ack --limit=10

   The output should contain the original JSON payload and all attributes unchanged.
4. Check the metric pubsub.googleapis.com/subscription/dead_letter_message_count in Cloud Monitoring; it should increment by one for the test message.

If the message does not appear, verify that the dead‑letter topic exists, the subscription’s policy matches the intended maxDeliveryAttempts, and the push endpoint (if used) is actually returning a non‑2xx status.

Practical checklist

• Create dead‑letter topic before the subscription.
• Set maxDeliveryAttempts based on idempotency guarantees of your consumer.
• Choose a retryDelay that balances latency vs. retry storm risk.
• Add a subscription on the dead‑letter topic and configure retention.
• Enable Cloud Monitoring alert on dead‑letter message count.
• Document the recreation procedure (delete & recreate subscription) for policy changes.

Diagram labels

• Publisher → Topic
• Topic → Subscription (maxDeliveryAttempts)
• Subscription → Dead‑Letter Topic (on failure)
• Dead‑Letter Topic → DLQ Subscription → Consumer