DEADLINE_EXCEEDED and grpc-timeout propagation in multi-hop gRPC services
0 reputation · 11 Dec 2024, 01:03 UTC
When a gRPC client sets a deadline, the remaining time is encoded in the grpc-timeout header and enforced across the RPC lifecycle. In a multi-hop service graph, the framework does not automatically chain that budget through independent downstream stubs, leaving teams to decide whether to propagate the original deadline or reserve a fresh budget per hop. Reusing a nearly expired deadline can turn one slow dependency into cascading DEADLINE_EXCEEDED errors, while reserving fresh budgets risks causing the overall call to exceed the client's original deadline. Retry configuration complicates the choice further, because retries may be permitted but the overall deadline still governs the call. A DEADLINE_EXCEEDED spike may also originate from queueing, network delay, or downstream latency rather than local handler execution, making it unclear which hop actually needs adjustment.
What is the most reliable way to decide between original deadline propagation and per-hop budget reservation when queue wait, handler time, and downstream latency are not yet separated? How should retry budgets be sized relative to the overall deadline without masking a downstream bottleneck? Which measurements are necessary before changing deadline or retry settings in a multi-hop gRPC deployment?
