Keras–TensorFlow concurrency: Can one shared model serve requests predictably?

Direct answer

No. Keras 3 on the TensorFlow backend publishes no bounded-latency contract for concurrent predict calls on one shared model instance. A shared instance can serve requests predictably only after you constrain inputs, warm up execution, pin thread and memory settings, and verify tail latency under load. Without those controls, p99 spikes are expected from thread-pool contention, retracing, accelerator scheduling, or request-handler overhead—not from model sharing itself.

Confirmed facts

• Keras 3 is the frontend; the TensorFlow runtime owns graph/eager execution, thread pools, devices, and accelerator scheduling.
• Keras 3 offers no service-level latency guarantee for concurrent inference on a shared instance.
• TensorFlow results do not transfer to the Keras 3 JAX or PyTorch backends.

Supported baseline (version assumption)

Use Keras 3 with a TensorFlow 2.x backend, not Keras 2 or TensorFlow 1.x. Pin one Keras 3 / TensorFlow 2.x minor pair verified together in staging. Exact minor versions cannot be certified here; record the pair in the test environment and treat it as a deployment unit.

Benchmark to separate backend contention from handler overhead

1. Warm up the shared instance with fixed-shape, fixed-dtype synthetic inputs until latency stabilizes.
2. Run the same inference volume sequentially, then concurrently with 8, 16, and 32 clients.
3. Record end-to-end p50, p99, and p999 plus throughput for each client count.
4. Instrument the request handler around the model call and add a no-op endpoint; handler overhead equals outer latency minus inference latency.
5. Use TensorFlow Profiler or TensorBoard traces to compare device/kernel time against queue or wait time under concurrency.

Interpretation

• If concurrent p99 rises while kernel time stays flat, contention is in request scheduling, thread pools, or queueing.
• If kernel or accelerator wait time rises, contention is backend-level.
• Vary tf.config.threading intra-op and inter-op threads only before model creation; settings are process-wide.

Verification

python -c "import keras, tensorflow as tf; print(keras.config.backend(), keras.__version__, tensorflow.__version__)"

Assumption: fixed input shapes and eager or compiled execution are used; variable shapes or unbounded automatic batching invalidate the baseline. Uncertainty: exact Keras 3 / TensorFlow 2.x minor versions and accelerator-specific behavior require current release verification.

Missing diagnostic that can change the recommendation: Is the target accelerator CPU or GPU? On GPU, enable memory growth before model initialization; on CPU, thread-pool sizing is usually the dominant tuning variable.