Keras Model Serving with TensorFlow Serving: Zero‑Downtime Model Update via Version Alias

Procedure for Zero-Downtime Model Updates

TensorFlow Serving manages model versions through a directory-based structure. To redirect a version alias (such as latest) to a new Keras model without interrupting active inference requests, follow these steps:

1. Stage the New Version: Place the new SavedModel directory—generated via model.save() in Keras—into the model base path. The directory must be named with an incrementing integer (e.g., /models/my_model/2/) and contain the saved_model.pb file and variables directory.
2. Update the Alias: Modify the version alias to point to the new version number. This is typically achieved by updating the model_config_list in the config.pbtxt file or by updating the symlink if the serving environment is configured for filesystem-based alias resolution.
3. Trigger Reload: TensorFlow Serving detects the configuration change. New incoming requests are routed to the new version, while requests already in flight continue to be processed by the previous version until completion.

Verification Steps

To confirm the alias has successfully shifted, query the model version endpoint:

curl http://localhost:8501/v1/models/my_model/versions

Verify that the latest alias now maps to the new version number. Additionally, check the server logs for a "Model restored" message, which indicates the new weights have been loaded into memory.

Unloading Previous Model Versions

It is safe to unload or delete the previous model version folder only after confirming that all in-flight predictions using that version have finished. Because TensorFlow Serving maintains the old version in memory until all active requests associated with it are resolved, immediate deletion of the source files can lead to errors if the server needs to reload a version during a crash or restart.

Assumptions and Uncertainty: This procedure assumes the server is configured with a model_config_list and that the filesystem permissions allow the serving user to read the new version directory. There is inherent uncertainty regarding the exact duration of "in-flight" requests; therefore, a grace period (e.g., several minutes) is recommended before deleting old version folders.

Diagnostic Detail Needed: To provide a more precise unload window, please specify if you are using a custom model_config_list or the default filesystem polling mechanism, as this changes how the server detects the alias update.