Should DEADLINE_EXCEEDED be retried for non-idempotent Google Cloud client library calls?

Answer

By default, non‑idempotent RPCs are not retried when they return DEADLINE_EXCEEDED. The generated retry configuration for most Google Cloud client libraries excludes this code for methods that can alter state (e.g., create, delete, update).

Why the default is “no”

• Server may have already applied the change. A deadline expiration only tells the client that it stopped waiting; it does not guarantee that the server aborted the operation.
• Duplicate side effects. Blindly retrying could create duplicate resources or corrupt data, violating consistency guarantees.
• Consistent library behavior. All major language clients (Java, Python, Go) expose a retryable‑code set that omits DEADLINE_EXCEEDED for non‑idempotent methods, ensuring a uniform developer experience.

How to verify the current policy in your code

1. Inspect the generated retry configuration. For example, in Java:

   MyServiceSettings settings = MyServiceSettings.newBuilder()
       .setRetryableCodes(MyServiceSettings.getDefaultRetryableCodes())
       .build();
   System.out.println(settings.getRetryableCodes());
   

   Look for the presence of DEADLINE_EXCEEDED in the returned set for the method you are calling.
2. Run a short‑deadline test. Set a per‑RPC deadline (e.g., 100 ms) and observe the server logs or response. If the client receives DEADLINE_EXCEEDED and does not retry automatically, the default policy is in effect.
3. Override the retry policy only if you have a safe idempotency guard. In Python you can do:

   from google.api_core import retry
   
   retry_policy = retry.Retry(
       predicate=retry.if_exception_type(),
       deadline=60.0,  # seconds
       initial=1.0,
       maximum=10.0,
       multiplier=2.0,
   )
   client.create_resource(request, retry=retry_policy)
   

   Ensure the service provides an idempotency key or transaction ID that can be reused.

When you might want to override the default

If the service exposes a mechanism (e.g., an X-Request-ID header or a transaction token) that lets you detect whether the operation was applied, you can safely retry a DEADLINE_EXCEEDED by adding that guard to your retry logic. Otherwise, keep the default non‑retry behavior to avoid duplicate side effects.

Missing diagnostic detail

Does the service you are calling provide an explicit way to confirm whether the operation completed (such as an idempotency key, a status endpoint, or a transaction ID)? If so, you can safely enable a retry on DEADLINE_EXCEEDED by incorporating that guard into your client code.