Pub/Sub subscription retry policy and duplicate write prevention
0 reputation · 31 Jul 2024, 01:22 UTC
0 reputation · 31 Jul 2024, 01:22 UTC
Pub/Sub subscription retry policy controls minimum and maximum backoff for redelivery after nack or ack deadline expiry. The configuration affects delivery timing, not delivery semantics.
The documented delivery guarantee is at-least-once. The API surfaces messageId per delivery, but there is no built-in deduplication window or correlation identifier provided by the service for retries. Applications are expected to implement idempotency for writes.
Behavior may differ across tiers and regions, and ack deadline settings interact with retry timing. The unresolved design point is whether deduplication on retry will remain entirely subscriber-side or become a configurable service behavior.
Does the current retry policy configuration expose any deduplication-related constraint or window? Is the responsibility for deduplication documented as a permanent subscriber-side requirement? Is a service-managed deduplicate-on-retry option under consideration?
33775 reputation · 31 Jul 2024, 05:07 UTC
No. The subscription retry policy exposes only redelivery backoff bounds such as minimumBackoff and maximumBackoff. It does not include a deduplication window, dedup key, duplicate-suppression flag, or retry correlation identifier. Yes, the documented delivery contract is at-least-once, which makes duplicate suppression a permanent subscriber-side responsibility. A service-managed deduplicate-on-retry option is not part of the documented retry policy; treat it as unavailable unless current product documentation for your exact Pub/Sub offering states otherwise.
Confirmed: Pub/Sub can redeliver a message after a nack, an expired ack deadline, a client restart, or an internal retry. The retry policy changes when a redelivery may occur, not whether duplicates are suppressed. Likely explanation for duplicate writes: a redelivered message is being processed as a new event. The service returns a messageId per delivery, but it does not remember what your subscriber already wrote, so the same business action can execute twice.
A short ack deadline can trigger redelivery while the first attempt is still running. That means concurrent duplicates are possible even without a nack. Do not use the ack deadline as a processing lock. For long-running handlers, extend the deadline or use lease management so the first attempt can finish, but still design for overlap.
messageId alone. Combine topic/subscription context and your own business identity if needed.maxDeliveryAttempts if you want to stop retry loops, but remember that exhausted messages can still be duplicates of earlier attempts.Create a throwaway subscription with explicit minimumBackoff and maximumBackoff. Publish one message, then nack it or let the ack deadline expire. Record delivery timestamps and message identifiers across several attempts to confirm backoff bounds and duplicate redelivery. Next, replay the same message identifier and business key twice against a test write path and confirm the second write is rejected or upserted without changing final state.
This answer assumes standard Cloud Pub/Sub semantics. Behavior can vary by product edition, subscription type, client library, region, and current release. Because retry-policy and delivery guarantees are version-sensitive, verify against the documentation for your exact target before relying on any deduplication behavior. If you are using a non-standard Pub/Sub product or a specialized subscription type, name it; that detail can change the recommendation.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.