Raising grpc.max_concurrent_streams above 100: latency spikes under bursty load on a single HTTP/2 connection

Context

A gRPC service currently runs with the default concurrent stream limit (100 in many implementations) over a single HTTP/2 connection per client. Under bursty concurrency, p99 latency rises sharply even though sequential-request benchmarks show flat, low latency. The plan under consideration is to raise grpc.max_concurrent_streams rather than add more client connections.

Constraints and uncertainty

Because all streams share one TCP connection, packet loss stalls every stream at once (head-of-line blocking), and the default HTTP/2 flow-control window (65,535 bytes per stream and per connection) can throttle many concurrent streams competing for the connection-level window. Raising the stream cap may reduce queuing at the limit but could worsen contention for the shared flow-control window and the server executor. Trade-offs of the alternative—client-side connection pooling—include higher resource use and uneven stream distribution. Behavior also differs across gRPC implementations and versions, so defaults should be confirmed for the version in use.

Questions

• Does increasing grpc.max_concurrent_streams on a single connection actually reduce concurrency-induced latency, or does it just shift the bottleneck to flow control and TCP head-of-line blocking?
• Which signals (e.g., GRPC_TRACE=http2,flowctl output, packet captures) best distinguish flow-control stalls from server thread-pool queuing?
• At what concurrency level does adding a second HTTP/2 connection outperform tuning a single one?