Managing Model Cold Starts and Scaling in KServe
Learn how to handle cold-start latency in KServe by balancing Knative autoscaling settings and decoupling model artifacts from serving images.
Loading…
ReadMeFeed / Community knowledge
#kserve
Real questions. Useful conversations. Find the people who know your stack.
kubeflowKubeflow Pipelines ↔ KServe: Zero‑Downtime Model Migration During Cluster Upgrade
Goal To migrate a small Kubeflow application from one cluster to another while keeping model inference traffic continuous, the migration must avoid any interruption during both the Pipelines control‑plane upgrade and the KServe model revision rollout. Constraints and Uncertainty Argo Workflows, the engine behind Kubeflow Pipelines, does not expose a document
KServe InferenceService: Default Concurrency Limits and Latency Under Concurrent Requests
Goal To evaluate how the default single‑threaded inference server (concurrency: 1) in Kubeflow KServe affects request latency when multiple inference requests are sent simultaneously. Constraints The inference server runs without the predictor.serving.kserve.io/concurrency annotation, inheriting the built‑in limit. The Kubernetes HPA does not scale pods auto
Kubeflow Pipelines and KServe: Zero-Downtime Model Migration Boundary
Kubeflow Pipelines and KServe: Zero-Downtime Model Migration Boundary Kubeflow Pipelines compiles workflow definitions into Argo Workflow YAML and executes PipelineRuns as immutable instances. KServe manages InferenceService deployments and relies on Istio for canary rollouts and traffic splitting. The integration boundary is the handoff from a completed tra
kubernetes