What memory overhead and latency impact occur when TensorFlow Serving loads a new model version while the old version remains active?

Question

Goal
Determine the additional memory consumption and request latency introduced when TensorFlow Serving loads a new model version while the previously served version remains active to handle in‑flight requests.
Context
TensorFlow Serving exposes models through REST and gRPC endpoints. The model_config file can specify a model_version_policy that governs whether the latest version is automatically served or a specific version is pinned. The server polls the filesystem for new versions at a configurable interval. During a version update, the old version may continue serving until the new version finishes loading, potentially resulting in both versions residing in memory simultaneously.
Uncertainty
The documentation does not quantify the memory overhead of keeping two versions loaded at once, nor does it specify the latency spike that clients may experience while the new version is being initialized and the old version is still handling traffic. This uncertainty is relevant for large models where the combined footprint could trigger out‑of‑memory conditions or cause noticeable delays.

What is the approximate extra memory usage when both the old and new model versions are resident in memory during the swap?
How does the latency of inference requests change for requests served by the old version versus those that begin to be routed to the new version during the transition?
Under what conditions (model size, polling interval, version policy) does the combined memory footprint risk exceeding the available RAM and causing an OOM event?

What memory overhead and latency impact occur when TensorFlow Serving loads a new model version while the old version remains active?

Goal

Context

Uncertainty

0 answers

Your perspective belongs here.

0 question comments