Using gRPC Deadlines to Stop Zombie Requests in Microservices

The Problem: Zombie Requests

\n

In a microservice call chain, a slow downstream service can cause upstream services to keep threads, memory, and sockets occupied even after the original client has given up.

\n

These \"zombie\" requests waste CPU and memory producing results that will be discarded.

\n\n

How gRPC Deadlines Work

\n

A gRPC deadline is an absolute point in time by which the RPC must finish. When a client sets a deadline, that timestamp is sent in the initial metadata. Each intermediate service subtracts its processing time and forwards the remaining deadline to the next hop.

\n

Deadline vs. Timeout

\n

A timeout is a duration (e.g., wait 5 seconds). A deadline is a fixed timestamp (e.g., finish by 10:00:05 AM). The key advantage is propagation: downstream services see the true time left, preventing the sum of individual timeouts from exceeding the client’s intent.

\n\n

Implementing Deadlines in Go

\n

Client-side

\n

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)\ndefer cancel()\n\nresp, err := client.GetUserData(ctx, &pb.UserRequest{Id: \"123\"})\nif err != nil {\n    if st, ok := status.FromError(err); ok && st.Code() == codes.DeadlineExceeded {\n        log.Printf(\"RPC deadline exceeded: %v\", err)\n    }\n}\n

\n\n

Server-side

\n

func (s *server) GetUserData(ctx context.Context, req *pb.UserRequest) (*pb.UserResponse, error) {\n    // Simulate work that may exceed the client's deadline\n    select {\n    case <-time.After(3 * time.Second):\n        return &pb.UserResponse{Data: \"done\"}, nil\n    case <-ctx.Done():\n        log.Println(\"Request cancelled by client; stopping work\")\n        return nil, ctx.Err()\n    }\n}\n

\n\n

Trade-offs and Limitations

\n

Setting a deadline too low turns normal latency spikes into errors, hurting availability. Setting it too high allows queued requests to exhaust thread pools during degradation, potentially causing cascading failures.

\n

Important limits:

\n

\n
• No automatic rollback: the call stops, but any side-effects already performed (e.g., DB writes) are not undone. Application-level cleanup or sagas are required.
\n
• Clock skew: because deadlines are absolute timestamps, large differences between client and server clocks can cause premature expiry or excess waiting. Keep clocks synchronized via NTP or similar.
\n

\n\n

Verification

\n

To confirm the behavior, add a synthetic delay in the server handler that exceeds the client’s deadline. Run the client and observe:

\n

\n
1. The client receives a status code of 4 (DEADLINE_EXCEEDED).
\n
2. The server logs show the context was cancelled and the long-running branch did not complete.
\n

\n

Check the logs for the \"Request cancelled by client\" line and verify no unnecessary work continues after the deadline.

\n\n

Closing

\n

Using gRPC deadlines gives you a lightweight way to enforce latency budgets across service boundaries. Pair them with proper server-side context checks and realistic deadline values to keep zombie requests from draining your resources.