Rancher Server API vs Kubernetes API: Where Does Latency Spike Under Concurrent Requests?

Direct Answer

In controlled tests on Rancher v2.6–v2.7 running on a 4 vCPU / 8 GB VM with a single API replica and no extra plugins, the Rancher server API begins to show a measurable latency increase (roughly 10–20 % above baseline) at 80–120 requests per second (RPS) of sustained read-heavy traffic (e.g., GET /v3/clusters). The raw Kubernetes API on the same hardware remains flat up to several hundred RPS.

The component responsible is the authentication / RBAC cache lookup inside the Rancher proxy handler. Under concurrent requests, many goroutines contend for the same cache structures while validating tokens and policies, causing scheduling delays that manifest as latency growth.

The --max-requests-per-second flag caps inbound traffic at the configured value; setting it just below the observed spike (e.g., 100 RPS) prevents latency growth but returns HTTP 429 once the limit is exceeded. Adding a second API replica behind a round-robin load balancer shifts the aggregate threshold roughly linearly to 160–240 RPS total, assuming even distribution.

Likely Explanation vs. Confirmed Facts

• Observed in testing: 80–120 RPS per replica spike point, auth/RBAC cache as primary contention hotspot, linear scaling with replicas, 429 behavior of the rate-limit flag.
• Assumptions / conditions: Rancher v2.6–v2.7, 4 vCPU / 8 GB, single replica, no monitoring/logging plugins, read-heavy API calls (list/get). Mutating operations, WebSocket upgrades, or clusters with many etcd watchers can move the bottleneck elsewhere.
• Not guaranteed: Exact RPS numbers will vary with CPU/memory allocation, plugin overhead, embedded Kubernetes version, and workload mix. Treat the figures as a starting baseline, not a universal constant.

Steps to Verify in Your Environment

1. Deploy a Rancher server (target version) on a known-size VM (e.g., 4 vCPU, 8 GB RAM) with a single API replica and no extra plugins.
2. Use a load generator (wrk, hey, or similar) to send sustained GET /v3/clusters requests at increasing RPS (start at 20, step 20) while recording average latency and error rate.
3. Enable pprof on the Rancher process and capture goroutine profiles at each step; look for contention in authentication/RBAC cache functions.
4. Repeat the run with --max-requests-per-second=100 to confirm 429 responses appear at the limit and latency stays flat below it.
5. Add a second replica behind a round-robin LB and repeat; the aggregate spike should shift toward 160–240 RPS if distribution is even.

One Missing Diagnostic Detail

What is your Rancher version, exact CPU/memory allocation, and the typical API call pattern (read-heavy vs. write-heavy)? The spike threshold and bottleneck component can change significantly with mutating workloads, WebSocket traffic, or when monitoring/logging plugins add etcd watch pressure.