Identifying HTTP/2 Flow‑Control Bottlenecks in gRPC

Question

When thousands of gRPC streams share a single connection, the HTTP/2 flow‑control window can be exhausted before the application payload is processed. The default per‑stream window is 64 KiB and the connection window is 1 MiB. If many streams compete, the sender must wait for WINDOW_UPDATE frames, adding round‑trip latency that may dominate the overall request‑response time.
The goal is to quantify the time spent waiting for WINDOW_UPDATE relative to actual payload handling, so we know whether flow‑control tuning or other optimisations will help. Constraints include memory usage per connection, interaction with keep‑alive and max‑message‑size settings, and the need to avoid out‑of‑memory errors when increasing the window.
Which metric should best capture the window‑exhaustion stalls? At what concurrency threshold does the default window become the limiting factor? Should we adjust the flow‑control window before enabling message compression or vice‑versa?

Tasadduq Burney · Answer

Short answerThe single most useful metric is per-connection time blocked waiting for WINDOW_UPDATE frames — i.e., the cumulative interval between the sender having data ready and receiving the window credit that lets it send. If that wait time is a large fraction of end-to-end latency, flow control is your bottleneck; if it is near zero, tuning the window will not help. With the default 64 KiB per-stream and 1 MiB connection window, the connection window becomes the limiting factor at roughly 16–32 concurrent large streams (1 MiB ÷ 64 KiB = 16 streams can each hold a full window in flight). And you should tune the window before enabling compression, because compression changes how many bytes each window unit buys you; tuning in the other order means re-measuring everything.What is confirmed vs. what is assumedConfirmed by the HTTP/2 spec (RFC 7540): the default initial per-stream window is 65,535 bytes, flow control is hop-by-hop per connection, and senders must stop when the window hits zero until a WINDOW_UPDATE arrives. The 1 MiB default connection window is a common gRPC implementation default, not a protocol constant — verify against your specific gRPC version (C-core, grpc-go, grpc-java expose different knobs, e.g. grpc.initial_window_size / grpc.initial_conn_window_size in C-core).Assumed, and worth verifying in your environment: that your streams actually fill the 64 KiB window. If your messages are small (a few KiB), per-stream windows never bind and only the connection window matters.Measuring the stallEnable transport tracing. For grpc-go: GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=2; for C-core: GRPC_TRACE=http2_flowctl GRPC_VERBOSITY=debug. Capture timestamps of WINDOW_UPDATE frames per connection.Run a load test sweeping concurrency from 1 to 64 streams with a fixed payload size (e.g., 256 KiB per message). Record per-request latency and WINDOW_UPDATE inter-arrival gaps.Plot stall time vs. concurrency. A knee in the curve near 16 concurrent full-window streams confirms the connection window is the limiter.Increase the connection window (e.g., 4 MiB) and rerun. A meaningful latency drop at the same concurrency confirms the diagnosis; no change means look elsewhere (CPU, max message size chunking, keepalive pings).Memory and safety constraintsWindow size is a memory commitment: the receiver must be prepared to buffer window-size bytes per stream and per connection. Quadrupling the window quadruples worst-case buffer demand per active stream. Under thousands of streams this is exactly how OOMs happen, especially if the application drains messages slowly. Raise windows incrementally (2×, then 4×), watch RSS under load, and pair any increase with bounded concurrency (max concurrent streams) so worst-case memory stays predictable. Keepalive settings are orthogonal but interact operationally: aggressive keepalive pings on an idle, window-starved connection can mask the stall in latency averages, so measure with keepalive fixed.Window tuning vs. compression: order mattersTune the window first, then evaluate compression. Flow control in HTTP/2 counts bytes on the wire, so compression effectively stretches the same window over more logical payload — but it adds CPU cost and can change message-size distribution. If you compress first, your window measurements reflect compressed bytes and any later compression change invalidates the tuning. Sequence: (1) measure stall time, (2) size windows to remove the stall within memory budget, (3) then enable compression and re-verify latency and CPU.One detail that changes the recommendationWhat is your average and p99 message size? If messages are under ~64 KiB, per-stream windows never bind and only the connection window and stream count matter — which changes both the threshold estimate and which knob to turn. Flag for review: exact default connection-window values and tuning APIs vary by gRPC implementation and version, so confirm against your stack's documentation before applying settings.

Identifying HTTP/2 Flow‑Control Bottlenecks in gRPC

1 answer

Short answer

What is confirmed vs. what is assumed

Measuring the stall

Memory and safety constraints

Window tuning vs. compression: order matters

One detail that changes the recommendation

Your perspective belongs here.

0 question comments