Diagnosing gRPC Health Check Failures: From UNAVAILABLE Errors to Root Cause

Your load balancer keeps pulling a service out of rotation even though the application itself is fine, or clients see intermittent UNAVAILABLE errors against the health service while server logs show nothing arriving at all. These are the two most common presentations of a broken gRPC health checking setup, and both are diagnosable in under an hour if you check things in the right order. This guide walks through the recognizable conditions, the likely causes, and the fixes tied to each finding.

What the gRPC health checking protocol actually requires

The standard health service (grpc.health.v1.Health) exposes a Check RPC that takes a service name string and returns a status: SERVING, NOT_SERVING, or UNKNOWN. Servers register the service name they want to report on (an empty string conventionally means "the server as a whole") and update status through the health servicer API as dependencies come up or go down. Most failures trace back to one of three places: the request never reaches the server, the server answers with the wrong status, or the RPC exceeds its deadline.

Symptom-to-cause quick reference

Step 1: Confirm the request reaches the server

If the server log shows no incoming health check requests, assume the RPC is dying in transit until proven otherwise. On the server host, capture traffic on the gRPC port (replace 50051 with yours):

sudo tcpdump -i any -nn port 50051 -c 50

You need root or CAP_NET_RAW to run this. You should see a steady cadence of small packets matching your configured health check interval. If nothing arrives, work outward: check security groups or firewall rules, then any L7 proxy in the path. A frequent culprit is an HTTP/1.1-only proxy or an L4 load balancer health-checking with TCP connects instead of real gRPC calls — the TCP probe succeeds while actual Check RPCs are never issued, or vice versa. If TLS is terminated at a proxy, confirm the proxy speaks HTTP/2 to the backend; gRPC does not run over HTTP/1.1.

Step 2: Verify the service name and status updates

When the server answers but always reports not-serving, the usual cause is a name mismatch. The health servicer keeps a per-service-name status map. If the server registered status under "" (empty) but the client asks for "payments.PaymentService", many implementations return NOT_FOUND rather than the overall status. Enable verbose logging to see exactly what is requested and returned:

# On both client and server processes, before startup
export GRPC_VERBOSITY=DEBUG
export GRPC_TRACE=http,health_check_client

These are environment variables read by the gRPC core library; they work for C-core-based implementations (Go, Python, Node, C++). Java uses its own logging (e.g., -Djava.util.logging.config.file with io.grpc.level=FINE). Expect to see the request path /grpc.health.v1.Health/Check and the response status in the trace. Then confirm the server actually updates status as dependencies change — a typical correct pattern in Go:

healthServer := health.NewServer()
healthServer.SetServingStatus("payments.PaymentService", healthpb.HealthCheckResponse_SERVING)
// On dependency loss:
healthServer.SetServingStatus("payments.PaymentService", healthpb.HealthCheckResponse_NOT_SERVING)

If SetServingStatus is only ever called once at boot, the health endpoint is decorative — it will report SERVING even when the database is gone. That is the fix for the "healthy health checks, failing traffic" row in the table.

Step 3: Rule out deadline and load problems

If health RPCs intermittently exceed their deadline, first check whether the deadline is simply too tight for your network (sub-second deadlines across availability zones are optimistic). Then check whether the health handler itself does real work. The Check RPC should read an in-memory status map and return — it must not query the database synchronously. If your implementation probes dependencies inline, a slow dependency turns every health check into a timeout, and the load balancer concludes the instance is dead precisely when it is overloaded. Move dependency evaluation to a background loop that calls SetServingStatus, and keep Check cheap.

Also sanity-check your check interval. Sub-second intervals across hundreds of clients can produce enough load to show up in CPU profiles, and worse, flapping intervals shorter than your client's timeout create oscillation: instance marked down, traffic stops, instance recovers, marked up, repeat.

Fixes mapped to findings

• No packets at the server: fix routing, firewall, or proxy HTTP/2 support. Re-run the packet capture to confirm arrival before touching application code.
• Name mismatch: align the client's requested service name with a name the server registered, or have the client use the empty-string convention for whole-server health.
• Stale status: wire SetServingStatus into real dependency lifecycle events (connection pool exhaustion, failed migrations, shutdown signals).
• Deadline exceeded: raise the client deadline to a measured value, and make the handler non-blocking.

When to escalate

Escalate to whoever owns the network path if packet capture shows the RPC leaving the client but never arriving, and you've ruled out your own security groups — that points at middlebox behavior you don't control. Escalate to the framework level (a gRPC issue tracker) only if you can reproduce a spec-conformant Check request getting a non-conformant response, captured with GRPC_TRACE logs attached.

Limitations and a practical verification pass

Behavior details here reflect the C-core and Go implementations that dominate production deployments; exact env-var trace names and NOT_FOUND-vs-UNKNOWN semantics differ slightly across language implementations and versions, so confirm against your version's docs. To verify the whole chain end to end after a fix: run grpcurl -plaintext -d '{"service":""}' host:50051 grpc.health.v1.Health/Check from the same network position as your real health checker, confirm SERVING, then force a dependency down and confirm the status flips to NOT_SERVING within one check interval. That last step is the one most teams skip, and it is the only one that proves the health signal means anything.