Custom deduplication header vs. inherent idempotency for Cloud Run retry safety

Short answer

Design the handler to be inherently idempotent. Under your stated constraint — the caller may redeliver the same logical message and the platform header X-Cloud-Run-Request-ID is not guaranteed stable across retries — an idempotent write path (PUT-style upserts keyed by a business identifier, or unique database constraints) is both more reliable and cheaper to operate than a custom deduplication token checked against a durable store.

Why the platform header is out

You've already identified the key fact: Cloud Run does not retry, and the request ID is generated per HTTP request, so a redelivery from Pub/Sub push or Cloud Tasks can arrive with a new value. Any dedup scheme keyed on that header silently fails exactly when you need it. This is well-established behavior, but confirm it for your trigger type by logging the header across a forced redelivery before ruling it out.

Comparing the two viable options

Inherent idempotency

Make the write itself safe to repeat:

• Upsert on a natural business key (INSERT ... ON CONFLICT DO NOTHING / DO UPDATE in Postgres, or Firestore set() with a deterministic document ID).
• Add a unique constraint so a duplicate insert fails cleanly and you treat that as success.
• Derive the key deterministically from the message payload (e.g., Pub/Sub message ID or an order ID), not from anything request-scoped.

Cost: no extra reads, no extra storage, no cleanup jobs. Failure modes are confined to your database's constraint enforcement.

Custom deduplication token

The caller attaches a UUID; the service records it in Firestore/Cloud SQL and rejects repeats. This works, but adds:

• An extra read+write on every request (latency and cost).
• A growing token store needing TTL/expiry policy.
• A subtle race: two concurrent deliveries can both pass the "not seen" check unless the token insert itself is atomic (e.g., insert-first with a unique constraint — at which point you've reinvented idempotency with extra steps).
• A hard dependency on the caller generating and propagating the token correctly.

Recommendation and caveats

Use inherent idempotency keyed on a stable business identifier, enforced by a database unique constraint. Reach for a token store only when the operation genuinely cannot be made idempotent (e.g., calling a non-idempotent downstream API) — and even then, implement the token check as an atomic insert, not check-then-write.

One caveat: Pub/Sub message IDs are stable across redeliveries of the same message, so if Pub/Sub is your only trigger, its message ID can serve as the idempotency key directly. If you also accept Cloud Tasks or direct HTTP, prefer a payload-derived key.

Verification

1. Deploy the endpoint, then publish the same Pub/Sub message twice (or force a nack/redelivery) and confirm exactly one row/document results.
2. Fire two concurrent identical requests (e.g., curl in parallel) to confirm the unique constraint handles the race.
3. Load-test at your expected QPS and compare write latency against a token-store variant if you're still weighing the two.

Assumption: your writes target a store supporting atomic upserts or unique constraints (Cloud SQL, Firestore, Spanner all qualify). If the write target is eventually consistent or lacks constraints, flag this for review — the recommendation shifts toward the atomic token-insert pattern.