k3s embedded API server and agent watch clients: who owns reconnection timing after a dropped watch?

Short answer

The agent’s client‑go library owns the reconnection timing, and k3s provides no supported flag or env‑var to change it. The backoff is compiled into the client‑go version bundled with the k3s release.

Confirmed vs. Likely

Confirmed by design: k3s agents use a standard client‑go client against the embedded API server. Watch reconnection logic is handled inside client‑go, not by any k3s setting. Patching the agent binary is unsupported and will break upgrades.

Likely, but verify for your release: The typical backoff floor is about 5 s and the cap around 30 s in recent client‑go releases. The exact initial delay, exponent, and maximum are defined by the specific client‑go version that k3s bundles, so check that version before assuming numbers.

Answers to the three specific questions

1. Any supported tuning mechanism?

No. k3s flags, environment variables, or kubeconfig fields do not alter the agent’s watch reconnection backoff. The kubeconfig only sets endpoints and credentials, not retry policy.

2. Floor or curve after a server‑side cancel?

When the API server cancels a watch context and closes the stream, client‑go treats the next attempt as a fresh watch. The backoff restarts from its minimum floor (≈5 s) rather than continuing the previous exponential climb. Only sustained failures (e.g., connection refused or timeouts) push the backoff toward the cap.

3. Do server‑side timeouts plus client backoff bound worst‑case staleness?

Yes, roughly. The maximum staleness for node and secret watches is bounded by the sum of the API server’s watch stream timeout (typically a few minutes) and the client‑go backoff at the moment of drop (capped near 30 s). Actual staleness depends on watch type and cluster load.

Verification steps for your release

1. Identify the bundled client‑go version:

   k3s --version
   # cross‑reference the k3s release’s go.mod for k8s.io/client-go
   

2. Observe the backoff sequence: start a watch, drop the connection (e.g., restart the server or block the port), and run the client with high verbosity:

   kubectl get nodes --watch --v=6
   # or a Go client with klog verbosity 2+ to log retry delays
   

3. Test floor vs. curve explicitly by creating a short‑lived watch with timeoutSeconds, letting the server close it, and measuring the interval before the client reconnects.

Practical recommendation

Accept the default curve and design around the bound: if short control‑plane blips are common, the primary lever is reducing those interruptions (e.g., HA servers, stable networking). The retry curve itself is not a tunable surface in k3s. If you need tighter staleness guarantees, validate the exact backoff and server timeout values on your specific k3s release rather than relying on generic numbers.