Resource exhausted OOM - GPU BFC allocator

Immediate Resolution

To resolve a ResourceExhaustedError caused by the BFC (Best-Fit with Coalescing) allocator, you must either reduce the memory footprint of your tensors or change how TensorFlow claims GPU memory. The most direct fixes are:

• Reduce Batch Size: Lower the batch size in your training or inference loop to decrease the size of the contiguous memory blocks requested.
• Enable Dynamic Memory Growth: Prevent TensorFlow from preallocating the entire GPU memory upon startup.

Implementation: Dynamic Memory Growth

Add the following configuration at the start of your script to allow the BFC allocator to request memory from the GPU as needed, rather than seizing the total available capacity immediately:

import tensorflow as tf

gpus = tf.config.list_physical_devices('GPU')
if gpus:
    try:
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
    except RuntimeError as e:
        print(e)

Technical Explanation

Confirmed Allocator Behavior

The BFC allocator manages GPU memory by maintaining lists of free memory blocks. By default, TensorFlow preallocates nearly all available VRAM to minimize the overhead of frequent system calls and to reduce fragmentation. An OOM error occurs when the allocator cannot find a contiguous block of memory large enough to satisfy a specific tensor request, even if the total sum of free memory across the GPU is sufficient.

Likely Causes and Trade-offs

While set_memory_growth(True) prevents the initial total seizure of VRAM—which is essential for multi-tenant environments—it introduces a different risk: fragmentation. Because memory is allocated incrementally, the BFC allocator may create a "checkerboard" of used and free blocks. Over time, this can lead to OOM errors that would not have occurred if a single, large contiguous block had been reserved at startup.

Verification Steps

1. Monitor VRAM: Run nvidia-smi -l 1 in a separate terminal to observe if memory usage climbs steadily until the crash or jumps to maximum immediately upon startup.
2. Baseline Test: Reduce the batch size by 50%. If the error disappears, the issue is a physical capacity limit. If the error persists, it is likely a fragmentation issue or a memory leak.

Diagnostic Detail Needed: Are you running multiple processes (e.g., multiple Jupyter notebooks or Docker containers) on the same GPU? If so, set_memory_growth is mandatory, as the default preallocation policy will cause any subsequent process to fail immediately.