KServe InferenceService: Default Concurrency Limits and Latency Under Concurrent Requests

Answer first

With the built-in default, KServe InferenceService processes one request at a time per pod. Under a 100-request concurrent burst that means requests queue behind the first, so per-request latency = base inference time + queue wait. Setting predictor.serving.kserve.io/concurrency to 8 allows up to eight requests to be processed in parallel per pod, which reduces queuing delay for the same burst, provided the GPU/CPU can sustain the parallelism.

Confirmed: the concurrency annotation exists and defaults to 1 for most KServe runtimes. HPA out of the box scales on CPU/memory, not on request latency or queue depth. Planned support for higher per-pod concurrency is documented as not fully implemented or tested in stable releases, so behavior is runtime-specific.

Confirmed facts vs likely explanation

Confirmed

• Default concurrency is 1 when the annotation is absent.
• KServe exposes request latency and resource metrics that can be scraped, but default HPA targets CPU utilization.
• GPU memory per active request is tied to model activations and runtime implementation; concurrency increases the number of active slots.

Likely, not verified

• Latency under 100 concurrent requests with concurrency 1 grows roughly with queue depth: ~ N * inference time. With concurrency 8 the queuing component is reduced, often toward ~ N/8 * inference time plus base inference time, assuming no GPU saturation.
• GPU memory rises with the number of active slots, roughly linear with concurrency if the runtime does not share activations. CPU utilization rises modestly from thread management and context switching.
• HPA will not react to latency increase from queuing unless a custom metric is configured.

What changes with GPU/CPU

Higher concurrency does not duplicate model weights, but it holds more activations and request contexts in parallel. If the GPU can execute eight streams without saturation, latency gains hold. If the GPU is already near capacity, raising concurrency can increase contention, power/thermal throttling, and even raise latency.

Minimal verification steps for this case

1. Deploy two identical InferenceServices, one without the concurrency annotation and one with predictor.serving.kserve.io/concurrency: "8".
2. Generate a 100-request concurrent burst with a tool such as hey or wrk, recording per-request latency from client timestamps or Prometheus histograms.
3. Collect GPU utilization and memory during the test, e.g., nvidia-smi logs, and CPU usage from node-exporter.
4. Observe HPA replica count and the metrics it is using. If replicas do not change while latency rises, CPU-based scaling is insufficient.

Missing diagnostic detail that changes recommendation: the runtime type and whether dynamic batching is enabled. Triton and TF Serving handle concurrency and batching differently from the KServe Python server, and dynamic batching can make concurrency 8 less beneficial or even harmful for latency.

Assumption: KServe stable release behavior as of 2026-10-18. This is a conservative draft pending current verification of your runtime and model.