Diagnosing gRPC UNAVAILABLE Errors: Step‑by‑Step Guide

Recognizable condition

A client invoking a remote procedure receives a gRPC status UNAVAILABLE (code 14) with a message similar to "Failed to connect to all addresses". This indicates that the channel could not establish a connection to any resolved endpoint.

Cause & diagnostic table

Cause	What to look for
Network connectivity	DNS failure, firewall blocking, or routing issue preventing reachability to the server IP:port.
TLS handshake problems	Expired/mismatched certificates, missing CA trust, or cipher‑suite mismatch causing the secure handshake to abort.
Server not running	Server process crashed, not started, or not bound to the expected port.
Load‑balancer misconfiguration	Health checks failing, empty backend pool, or mis‑routed traffic.
Client deadline too short	RPC deadline expires before the server can respond, especially under high latency or transient load.

Ordered checks

1. Verify basic network reachability

   Run from the client host:

   ping 

   If ping is blocked, try a TCP connect:

   telnet  

   Required permissions: normal user can run ping/telnet; no special privileges needed. Risk: none beyond normal network traffic.

   Expected check: ping returns replies; telnet either connects (shows a blank screen or "Connected") or fails quickly with "Connection refused" or timeout.

   Limitation: ICMP may be blocked while TCP works; rely on telnet for definitive port‑level test.

2. Confirm DNS resolution

   nslookup 

   Ensure the returned A/AAAA records match the IP you tested.

   Permissions: normal user. Risk: none.

3. Inspect TLS credentials (if using secure gRPC)

   Check certificate expiry and trust chain:

   openssl s_client -connect : -servername  -showcerts

   Look for "Verify return code: 0 (ok)". Any non‑zero code indicates a trust issue.

   Permissions: normal user. Risk: exposing the certificate chain in logs; avoid printing to shared logs in production.

4. Query the gRPC Health service (if enabled)

   Using grpcurl (or any health‑checking client):

   grpcurl -plaintext : grpc.health.v1.Health/Check

   Expected JSON response: {"status": "SERVING"}. If the service is not enabled, you will get UNIMPLEMENTED.

   Permissions: normal user; requires grpcurl installed. Risk: none.

5. Check load‑balancer health (if applicable)

   Consult the LB’s health endpoint or logs; verify that backends are marked healthy and registered.

   Permissions: depends on platform; often read‑only access to monitoring APIs is sufficient.

6. Temporarily increase client deadline

   In the client code or config, set a higher timeout, e.g., 30 seconds:

   // Example in Go
   ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
   defer cancel()
   resp, err := client.MyMethod(ctx, req)

   Re‑run the RPC. If it succeeds, the original deadline was too low for observed latency.

   Permissions: code change; requires rebuild/redeploy. Risk: masking latency issues; do not leave the deadline excessively high in production.

7. Review server logs for startup or binding errors

   Look for messages like "failed to listen on port X" or panic stacks.

   Permissions: access to server log files (often requires admin or service account).

Fixes tied to findings

• Network connectivity: correct DNS entries, open required firewall ports (typically the gRPC port, e.g., 50051), verify routing tables.
• TLS handshake: renew or replace expired certificates, ensure the client trusts the server’s CA, align cipher suites (e.g., enable TLS 1.2+ suites).
• Server not running: start or restart the server process; confirm port binding with netstat -tlnp | grep (requires root or equivalent to see PID).
• Load‑balancer misconfiguration: adjust health‑check thresholds, ensure backends are healthy and registered, verify that the LB forwards to the correct port.
• Client deadline too short: raise the deadline to accommodate measured latency (add a safety margin), or implement retry with exponential backoff.
• General visibility: enable verbose gRPC logging for deeper insight:

  export GRPC_VERBOSITY=DEBUG
  export GRPC_TRACE=all

  Capture logs on both client and server; do not leave these env vars enabled in high‑traffic production due to log volume.

Escalation criteria

If after completing all checks and applying the corresponding fixes the UNAVAILABLE error persists:

1. Capture a full trace:

   export GRPC_VERBOSITY=DEBUG
   export GRPC_TRACE=all
   # run the failing RPC again

2. Collect client and server logs (including the trace output).
3. Escalate to the platform/network team or vendor support, providing the trace, logs, and a summary of steps already taken.

Verification

• Re‑run the failing RPC; the client should receive status OK (code 0) with the expected payload.
• Server‑side logs should show the request processed without error.
• If a health service is used, grpc.health.v1.Health/Check returns {"status": "SERVING"}.
• When a load balancer is present, its health endpoints report all backends as healthy and traffic is distributed.