Handling Poison Pill Messages with Pub/Sub Dead Letter Topics

Preventing Infinite Retry Loops in Pub/Sub

A "poison pill" is a message that is syntactically correct enough to be delivered but contains data that causes your subscriber application to crash or return an error every time it is processed. Without a strategy to remove these messages, Pub/Sub will continue to redeliver them indefinitely, wasting compute resources, inflating logs, and potentially blocking the processing of healthy messages.

The solution is a Dead Letter Topic (DLT). A DLT allows you to define a maximum number of delivery attempts; once that threshold is reached, Pub/Sub automatically moves the problematic message to a separate topic for manual inspection or separate processing, rather than returning it to the original subscription queue.

Prerequisites

• A Google Cloud project with the Pub/Sub API enabled.
• An existing Pub/Sub topic and subscription (Push or Pull).
• roles/pubsub.editor or roles/owner permissions to modify subscription settings and create topics.

Implementation Procedure

1. Create the Dead Letter Topic

First, create a dedicated topic to hold the failed messages. It is best practice to name this topic consistently with the original, such as orders-topic-dead-letter.

# Run in Cloud Shell or local terminal with gcloud installed
gcloud pubsub topics create orders-topic-dead-letter

2. Configure Service Account Permissions

Pub/Sub does not use your user identity to move messages; it uses a system-managed service account. You must grant this account permission to publish to the DLT and to acknowledge messages from the original subscription.

Find your project number first:

gcloud projects describe [PROJECT_ID] --format="value(projectNumber)"

Grant the required roles to the Pub/Sub service account (formatted as service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com):

# Grant permission to publish to the dead letter topic
gcloud pubsub topics add-iam-policy-binding orders-topic-dead-letter --member="serviceAccount:service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com" --role="roles/pubsub.publisher"

# Grant permission to acknowledge messages from the original subscription
gcloud pubsub subscriptions add-iam-policy-binding orders-subscription --member="serviceAccount:service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com" --role="roles/pubsub.subscriber"

3. Attach the Dead Letter Policy to the Subscription

Update the subscription to route messages to the DLT after a specific number of failed attempts. The max-delivery-attempts must be between 5 and 100.

gcloud pubsub subscriptions update orders-subscription --dead-letter-topic=orders-topic-dead-letter --max-delivery-attempts=5

Verification and Testing

To verify the configuration, you must simulate a failure that exceeds the delivery threshold.

1. Inject a Failure: Publish a message to the main topic that you know will cause your subscriber to return a non-success code (e.g., a message missing a required JSON field).
2. Monitor Delivery: Observe the subscriber logs. You should see the same message delivered 5 times (matching your max-delivery-attempts).
3. Check the DLT: Create a temporary subscription to the dead letter topic and pull messages to confirm the poison pill has arrived.

   gcloud pubsub subscriptions create dlt-test-sub --topic=orders-topic-dead-letter
   gcloud pubsub subscriptions pull dlt-test-sub --auto-ack

Comparison: DLT vs. Standard Retries

Feature	Standard Retry	Dead Letter Topic
Behavior	Retries until acknowledged or expires.	Moves to DLT after N attempts.
Resource Impact	High (CPU/Log waste on poison pills).	Low (Problematic messages are isolated).
Visibility	Hidden in the main queue.	Explicitly visible in a separate topic.
Metadata	Preserves original publish time.	New publish time assigned upon move.

Limitations and Risks

• Metadata Loss: When a message is moved to a DLT, it is effectively republished. The original publish_time is lost; the message will have a new timestamp reflecting when it entered the DLT.
• Transient Errors: If max-delivery-attempts is set too low (e.g., 5), a brief network flicker or a database restart might accidentally move healthy messages to the DLT.
• IAM Failures: If the Pub/Sub service account lacks roles/pubsub.publisher on the DLT, the message will not be moved and will continue to retry in the original subscription, potentially causing the very loop you intended to prevent.

Rollback

To remove the dead letter policy and return to standard retry behavior, update the subscription to remove the topic association:

gcloud pubsub subscriptions update orders-subscription --clear-dead-letter-topic