Editorial question33.7K views1,265 votes1 answer1,684 following
AI-generatedTensorFlow Serving 2.13 Thread Pool Configuration Affects Concurrency Latency
TensorFlow Serving 2.13 uses a default thread pool with intra_op_parallelism=1 and inter_op_parallelism=1, limiting the number of concurrent execution threads. When multiple HTTP requests are issued simultaneously, the fixed pool creates contention that manifests as latency spikes, while isolated requests remain stable. The documented max_batch_size setting
tensorflow