Why gRPC's PickFirst Policy Can Break Your Multi-Instance Deployment (And How to Spot It)

The Hidden Cost of Default Load Balancing

When you spin up a gRPC client pointing to multiple backend instances, you might assume requests distribute evenly. They won't. The default PickFirst policy silently picks one server and sticks with it until that connection dies—leaving your other instances idle and your system vulnerable to single points of failure.

How PickFirst Actually Works

PickFirst is a client-side load balancing policy that selects the first resolved address and establishes a single connection. It does not rotate through servers, does not perform health checks, and will not reconnect to an alternative if the primary becomes unhealthy—only when the TCP connection itself drops. This behavior is built into the Go and Java gRPC SDKs as the default for most configurations.

Configuration Differences Across Languages

In Go, you must explicitly set the load balancer using the WithBalancer dial option:

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/balancer/roundrobin"
)

conn, err := grpc.Dial(
    "localhost:50051",
    grpc.WithBalancer(roundrobin.New),
    grpc.WithInsecure(),
)

In Java, the policy is controlled via the io.grpc.LoadBalancerProvider and can be overridden at channel creation:

ManagedChannel channel = ManagedChannelBuilder
    .forAddress("localhost", 50051)
    .defaultLoadBalancingPolicy("round_robin")
    .usePlaintext()
    .build();

Testing Your Load Balancer Policy

Use grpcurl to verify which policy is active:

grpcurl -plaintext -import-path . -proto your.proto localhost:50051 list

If you're connecting to multiple endpoints via DNS or a service discovery mechanism, observe connection patterns. With PickFirst, all traffic should route through a single IP address until that connection fails.

Trade-Offs and Limitations

PickFirst excels in scenarios with a single backend or where failover is handled externally (e.g., via a service mesh). However, it provides no load distribution, meaning uneven resource utilization across your server fleet. Additionally, since PickFirst doesn't implement health checking, a server that's CPU-starved or hung can continue receiving all traffic until the network connection times out—potentially causing cascading failures if the client doesn't implement its own health monitoring.

Actionable Next Steps

1. Audit your client configurations: Check if any gRPC clients rely on defaults without specifying a load balancer.
2. Test failover behavior: Simulate a server crash and measure recovery time and traffic redistribution.
3. Implement health checking: If using PickFirst, add application-level health checks or consider switching to round_robin with a health-checking proxy.
4. Monitor connection state: Use gRPC's connection state APIs to log when connections transition between Ready, Connecting, and TransientFailure states.

In multi-instance deployments, PickFirst is rarely the right choice. The cost of misconfiguration is silent underperformance and unexpected downtime during partial outages.