gRPC max concurrent streams per connection: tuning stream limits versus opening additional connections

1. Choosing a Max Concurrent Streams Value

When a gRPC client or server opens an HTTP/2 connection it advertises a SETTINGS_MAX_CONCURRENT_STREAMS value. The peer accepts the smaller of the two values, so the effective limit cannot exceed the peer’s advertised number. Typical defaults in recent stable releases are:

• Go – 1000 streams
• Java – 200 streams
• C‑core – 1000 streams

Because the limit is per connection, a single connection will queue any RPCs that exceed the negotiated value until a stream slot frees up. If your workload consistently approaches this ceiling, the queue latency will grow.

In practice the choice between raising the stream limit and opening more connections is empirical. Start with the default, measure queue latency, and only raise the limit if:

• The average grpc_server_streams_pending latency is below 5–10 ms of your target.
• Flow‑control windows (http2_connection_flow_control_window and http2_stream_flow_control_window) are not exhausted.
• Server thread‑pool or GC metrics do not spike when the limit is increased.

If any of these conditions are violated, consider adding a new connection instead.

2. When to Open Additional Connections

Opening a new connection is beneficial when the queue latency for pending streams consistently exceeds the SLA threshold. The following rule of thumb works well for most workloads:

1. Measure grpc_server_streams_pending and grpc_server_handling_time while keeping the stream limit at the negotiated value.
2. If the average pending‑stream latency > 5–10 ms and the overall RPC latency rises, start opening a second connection.
3. Verify that grpc_client_connections_open (client side) or grpc_server_connections_open (server side) shows the new connection being established.
4. Re‑measure. If the latency drops to the target range, keep the connection count. If not, either raise the stream limit (within the peer’s advertised cap) or add another connection.

Remember that each new connection incurs TLS handshake overhead and consumes file descriptors, so keep the total number of connections reasonable. In most cases 2–4 connections per logical endpoint suffice for bursty workloads.

3. Metrics to Distinguish Stream‑Limit Queuing from Flow‑Control Starvation

Metric	What It Shows
grpc_server_streams_pending	Number of RPCs waiting for a stream slot.
grpc_server_handling_time	Time spent in user code; spikes may indicate thread‑pool pressure.
http2_connection_flow_control_window	Remaining bytes the peer can send on the connection.
http2_stream_flow_control_window	Remaining bytes the peer can send on a particular stream.
grpc_client_connections_open	Number of active client connections.
grpc_server_connections_open	Number of active server connections.

When grpc_server_streams_pending climbs while the flow‑control windows stay well above zero, you’re limited by the stream count. Conversely, if the flow‑control windows deplete rapidly while the pending‑stream count is low, payload size and flow‑control are the bottleneck.

Key Takeaway

Start with the negotiated stream limit, monitor queue latency and flow‑control windows, and only raise the limit or add connections when the metrics indicate a clear constraint. This empirical path ensures you avoid unnecessary thread‑pool pressure or resource waste.

Diagnostic Question

What is your target per‑RPC latency or SLA? Knowing the acceptable latency helps set the queue‑latency threshold that triggers opening a new connection.