Choosing Between gRPC Built‑in Retry and a Custom Interceptor

Decision at a Glance

When a gRPC client fails repeatedly, you can either rely on the protocol’s built‑in retry policy or implement a custom retry interceptor. The choice hinges on server support, idempotency requirements, code maintenance, and deadline handling.

Constraints to Consider

• Server advertisement: Built‑in retry only works if the server’s service config lists retryable error codes for the method.
• Idempotency: By definition, the built‑in policy retries only idempotent RPCs; non‑idempotent calls are ignored.
• Language support: Some gRPC libraries (e.g., Go 1.23+, Java 1.19+) expose built‑in retry; older versions require manual handling.
• Deadline propagation: Retries must respect the client‑side deadline; mis‑configuring backoff can cause double timeouts.
• Metrics and tracing: Built‑in retry is opaque; custom interceptors can emit per‑retry metrics and trace spans.

Option Comparison Table

Feature	Built‑in Retry	Custom Interceptor
Server‑side config required?	Yes (service config JSON)	No
Idempotency enforcement	Automatic – only idempotent methods retried	Developer‑defined
Language agnostic?	Yes – part of the gRPC spec	No – implementation per language
Code complexity	Low – declarative config	High – interceptor logic, backoff, error handling
Deadline handling	Built‑in respects client deadline but may cause double timeouts	Full control – can abort immediately when deadline expires
Metrics & tracing integration	Limited – opaque retries	Full – emit logs, counters, spans per retry
Risk of duplicate side effects	Low – only idempotent methods retried	High – must guard against non‑idempotent retries
Version dependency	Requires recent gRPC library (e.g., Go 1.23)	Independent of library version

Trade‑offs Explained

Built‑in retry is a zero‑code, declarative approach that works across languages. It’s perfect when you control both server and client and the RPCs are naturally idempotent. However, you lose visibility into each retry attempt, and you cannot tailor the logic for non‑idempotent calls or integrate detailed metrics without additional tooling.

Custom interceptor gives you full command over when to retry, which error codes to treat as transient, and how to back off. This flexibility is invaluable for legacy services that don’t expose retryable codes or for non‑idempotent operations where you need fine‑grained safety checks. The downside is increased code churn, the need to manage deadlines yourself, and the risk of accidental duplicate side effects if the interceptor logic is not carefully designed.

Concrete Validation Example

Scenario

We have a local gRPC server exposing a method GetStatus that always returns UNAVAILABLE to simulate a transient failure. We want to confirm that both retry mechanisms perform exactly two retries before giving up, respecting a 200 ms client deadline.

Built‑in Retry Configuration

# service_config.json
{
  "methodConfig": [
    {
      "name": [
        {"service": "example.StatusService", "method": "GetStatus"}
      ],
      "retryPolicy": {
        "maxAttempts": 3,
        "initialBackoff": "100ms",
        "maxBackoff": "1s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": ["UNAVAILABLE"]
      }
    }
  ]
}

Run the client with the JSON config loaded (e.g., --grpc-service-config=service_config.json). The client will automatically retry the RPC twice, each time waiting 100 ms, 200 ms, then fail after the third attempt. The total elapsed time should be roughly 300 ms plus the RPC processing time.

Custom Interceptor Implementation (Go)

package main

import (
    "context"
    "log"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)

func retryInterceptor(maxAttempts int, backoff time.Duration) grpc.UnaryClientInterceptor {
    return func(
        ctx context.Context,
        method string,
        req, reply interface{},
        cc *grpc.ClientConn,
        invoker grpc.UnaryInvoker,
        opts ...grpc.CallOption,
    ) error {
        var err error
        for attempt := 1; attempt <= maxAttempts; attempt++ {
            err = invoker(ctx, method, req, reply, cc, opts...)
            if err == nil {
                return nil
            }
            if s, ok := status.FromError(err); ok {
                if s.Code() != codes.Unavailable {
                    return err
                }
            }
            if attempt == maxAttempts {
                break
            }
            log.Printf("Retry attempt %d for %s", attempt, method)
            select {
            case <-time.After(backoff):
                backoff *= 2
            case <-ctx.Done():
                return ctx.Err()
            }
        }
        return err
    }
}

func main() {
    conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithUnaryInterceptor(retryInterceptor(3, 100*time.Millisecond)))
    if err != nil {
        log.Fatalf("dial error: %v", err)
    }
    defer conn.Close()

    client := NewStatusServiceClient(conn)
    ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
    defer cancel()

    _, err = client.GetStatus(ctx, &GetStatusRequest{})
    if err != nil {
        log.Printf("final error: %v", err)
    }
}

In this interceptor, we attempt the RPC up to three times, doubling the backoff each retry. The ctx.Done() channel ensures we abort if the client deadline expires before the retry loop completes.

Validation Steps

1. Run the server that always returns UNAVAILABLE on GetStatus.
2. Execute the built‑in retry client and observe that the RPC is attempted three times. The total time should be close to 300 ms.
3. Execute the custom interceptor client. Check the logs for two "Retry attempt" entries and a final error. Verify the elapsed time is similar.
4. Set the client deadline to 200 ms. Both approaches should abort early, returning DEADLINE_EXCEEDED if the deadline is hit before the retry loop completes.
5. Replace GetStatus with a non‑idempotent method (e.g., CreateItem). With built‑in retry, no retry occurs. With the custom interceptor, add a guard that skips retries for non‑idempotent methods.

Practical Checklist Before Deployment

• Confirm the gRPC library version supports built‑in retry (e.g., Go 1.23+, Java 1.19+).
• Ensure server service config advertises the correct retryable status codes.
• Validate that the retry policy respects the client deadline; adjust maxAttempts or backoff if necessary.
• If using a custom interceptor, instrument logs or counters to confirm the number of retries and that no duplicate side effects occur.
• Run end‑to‑end tests under load to measure overall latency impact.

Conclusion

Use the built‑in retry policy when you have control over the server, your RPCs are idempotent, and you prefer a declarative, language‑agnostic solution. Opt for a custom interceptor when you need granular control over retry conditions, must handle non‑idempotent calls safely, or require detailed telemetry. Both approaches can coexist; for example, you might use built‑in retry for core services and a custom interceptor for legacy or third‑party APIs.