KServe InferenceService: Default Concurrency Limits and Latency Under Concurrent Requests

Goal

To evaluate how the default single‑threaded inference server (concurrency: 1) in Kubeflow KServe affects request latency when multiple inference requests are sent simultaneously.

Constraints

The inference server runs without the predictor.serving.kserve.io/concurrency annotation, inheriting the built‑in limit. The Kubernetes HPA does not scale pods automatically based on request concurrency, so increased load will likely raise per‑request latency rather than pod count.

Unresolved Decision

The KServe documentation notes that supporting higher per‑pod concurrency (e.g., concurrency: 8) is a planned feature but has not been fully implemented or tested in stable releases. This raises questions about the optimal configuration for latency‑sensitive workloads.

Specific Questions

• What measurable latency differences occur when the concurrency annotation is set to 8 compared to the default 1 under a 100‑request concurrent load?
• Does enabling higher concurrency alter GPU memory consumption or CPU utilization in a way that could negate latency gains?
• Will the current HPA metrics for InferenceService pods adapt to increased concurrency, or is manual scaling required to maintain acceptable latency?