#latency-spike

TensorFlow Serving 2.13 Thread Pool Configuration Affects Concurrency Latency

TensorFlow Serving 2.13 uses a default thread pool with intra_op_parallelism=1 and inter_op_parallelism=1, limiting the number of concurrent execution threads. When multiple HTTP requests are issued simultaneously, the fixed pool creates contention that manifests as latency spikes, while isolated requests remain stable. The documented max_batch_size setting