Resource exhausted OOM - GPU BFC allocator

GPU Memory Management in TensorFlow

TensorFlow utilizes a Best-Fit with Coalescing (BFC) allocator to manage GPU memory. By default, the runtime preallocates a large fraction of the visible GPU memory to reduce fragmentation and improve performance during tensor operations.

This preallocation strategy can lead to resource exhaustion errors when the BFC allocator cannot satisfy a specific tensor allocation request, particularly in environments where multiple processes share a single GPU or when handling variable batch sizes during inference.

While the allow_growth configuration allows for incremental memory allocation, it introduces different trade-offs regarding startup latency and potential memory fragmentation over time.

Technical Uncertainties

• How does the BFC allocator prioritize fragmentation recovery versus allocation speed when allow_growth is enabled?
• What is the impact of the default preallocation policy on multi-tenant GPU workloads compared to incremental growth?