Does the Kubernetes API server end a watch stream promptly when the client cancels its context?

Short Answer

On HTTP/2 (the default for Kubernetes 1.34–1.37), cancelling the client-go context sends an RST_STREAM frame that the Go http2 server translates into a request-context cancellation. The watch handler stops within milliseconds, without waiting for the next event. On HTTP/1.1 there is no in-band signal; the server only notices when it tries to write the next event or when --min-request-timeout (default 30 minutes) expires. This transport-level difference is the dominant factor; the Kubernetes version in the 1.34–1.37 range does not change the behavior.

Transport Behavior Detail

HTTP/2 (default)

• Client-go calls context.CancelFunc → the HTTP/2 client sends RST_STREAM (error code CANCEL or NO_ERROR).
• The Go standard library's HTTP/2 server receives the frame and cancels the http.Request.Context() that the API server's watch handler is already observing.
• The handler's next select on that context returns, the watch is removed from the watch cache, and the goroutine exits. Typical latency is sub-millisecond to a few milliseconds, bounded only by how quickly the server's event loop processes the cancellation.

HTTP/1.1 (fallback or forced)

• No protocol-level cancellation frame exists.
• The API server continues to hold the watch handler until it attempts to write the next event (which fails with a broken-pipe error) or the connection idles past --min-request-timeout.
• Teardown can therefore take anywhere from the next event interval up to 30 minutes.

Version Consistency (1.34–1.37)

The watch handler has used request.Context() for cancellation since before 1.34. No release in the 1.34–1.37 window introduced changes to this path. Aggregated API servers and the --watch-cache-sizes flag forward the same request context, so they do not add measurable latency.

Proxy and Load-Balancer Caveat

If a proxy (cloud LB, kube-proxy in certain modes, service mesh sidecar) terminates HTTP/2 and re-originates HTTP/1.1 to the API server, the prompt RST_STREAM behavior is lost. The API server then behaves as if the client spoke HTTP/1.1. Verify the end-to-end protocol with a packet capture on the API server's secure port.

Measuring Teardown Latency in a Test Cluster

1. Deploy a kind or single-node cluster with the API server flag --vmodule=watch*=5 (or --v=5 filtered to watch logs) to emit timestamped "watch handler started/stopped" lines.
2. Run a client-go watch with a short-lived context (e.g., context.WithTimeout(ctx, 500*time.Millisecond)).
3. Correlate the client cancel time (log it in the client) with the server's "watch handler stopped" log entry.
4. Repeat with --http2-max-streams-per-connection=1 on the API server to force HTTP/1.1 and observe the longer tail.
5. Capture traffic on the API server port (tcpdump -i any port 6443 -w watch.pcap) and confirm RST_STREAM arrives before the handler-stop log line.
6. Connect directly to the API server's secure port (bypass kube-proxy/LB) so proxy buffering does not contaminate the measurement.

Practical Recommendation

Application code can rely on prompt server-side teardown if the full path is HTTP/2. Operators should still size timeouts for the HTTP/1.1 worst case (next event write or --min-request-timeout) unless they have verified HTTP/2 all the way to the API server. Lingering watch handlers after a client cancel are normal on HTTP/1.1 and should not be treated as a leak.

One Diagnostic Detail That Changes the Recommendation

Confirm whether your cluster's ingress path (load balancer, API server --http2-max-streams-per-connection setting, or service mesh) preserves HTTP/2 to the API server. If it does not, plan for HTTP/1.1 teardown latency.