Raising grpc.max_concurrent_streams above 100: latency spikes under bursty load on a single HTTP/2 connection

Direct Answer

Raising grpc.max_concurrent_streams alone rarely reduces tail latency under bursty load on a single HTTP/2 connection. It removes the application-level queue at the stream limit but shifts contention to the shared connection flow-control window (default 64 KiB) and the server executor pool. TCP head-of-line blocking remains a fundamental limitation: any packet loss stalls every stream simultaneously. Adding a second HTTP/2 connection typically outperforms single-connection tuning once sustained concurrency exceeds roughly 2× the bandwidth-delay product (BDP) in streams.

Confirmed Behavior

• Flow-control contention: With many active streams, the 64 KiB connection-level window is exhausted quickly. Each stream waits for WINDOW_UPDATE frames, adding an RTT per round-trip. This serializes progress and inflates p99 latency.
• Head-of-line blocking: All streams share one TCP connection. Packet loss blocks every stream until retransmission completes, regardless of stream count.
• Implementation defaults differ: Go (100 streams, 64 KiB window, auto-tunes to 16 MiB since v1.50), Java/Netty (100 streams, 64 KiB stream window, 1 MiB connection window), C++ Core (100 streams, 64 KiB both). Verify your runtime version.

Diagnostic Signals That Distinguish Bottlenecks

1. GRPC_TRACE=http2,flowctl (client and server): Frequent zero-window pauses and rapid WINDOW_UPDATE frames indicate flow-control stalls. Correlate zero-window duration with p99 spikes.
2. Executor queue depth: Java ThreadPoolExecutor.getQueue().size(), Go runtime/pprof goroutine profile, C++ completion queue depth. Low queue depth with high zero-window time = flow control. High queue depth = executor saturation.
3. Packet capture (tcpdump/Wireshark): TCP retransmissions correlating with latency spikes = HOL blocking. Clean ACKs but delayed DATA frames = flow control.

Recommended First Steps

1. Run a controlled burst test (e.g., ghz or custom client) at 50, 100, 200, 500 concurrent RPCs on a single connection. Capture p50/p99/p999 latency, GRPC_TRACE=http2,flowctl logs, and tcpdump.
2. If zero-window time > 50% of RPC latency, increase SETTINGS_INITIAL_WINDOW_SIZE (e.g., to 1 MiB) before raising the stream limit. Verify peer support via SETTINGS frames in the capture.
3. If executor queue depth spikes, tune the server thread pool.
4. Repeat the burst test with 2 and 4 client connections (same total concurrency) to measure the crossover point. Compare p99 at each level.

When to Add Connections

For a typical 10 Gbps LAN with 0.5 ms RTT (BDP ≈ 625 KB), a single 64 KiB window supports ~10 streams. Bursty loads exceeding 20–30 concurrent RPCs benefit from connection pooling. The crossover scales with RTT and window size. Client-side pooling (e.g., Java ManagedChannelBuilder, Go grpc.WithDefaultCallOptions, .NET HttpClientHandler.PooledConnectionLifetime) distributes streams across TCP connections, eliminating single-connection HOL blocking and flow-control contention at the cost of ~2–3× more file descriptors and TLS handshakes.

One Missing Detail

What is your gRPC runtime and version (e.g., gRPC-Go v1.62, gRPC-Java 1.62, gRPC-C++ 1.62)? Auto-tuning behavior and default window sizes vary, and some proxies clamp SETTINGS_INITIAL_WINDOW_SIZE to 64 KiB end-to-end.