Diagnosing TensorFlow GPU Visibility Failures When nvidia-smi Sees the Device

The symptom: nvidia-smi sees a GPU, TensorFlow does not

You run nvidia-smi and see your accelerator listed, but tf.config.list_physical_devices('GPU') returns an empty list. Alternatively, TensorFlow imports cleanly but throws CUDA or cuDNN errors when you execute an operation, or every operation silently runs on CPU with a severe slowdown. This is a device-visibility or runtime packaging problem, not a bug in your model or dataset.

The useful takeaway: you can isolate the failure in minutes by comparing what the NVIDIA driver sees, what the TensorFlow build was compiled against, and what environment variables are hiding from the process.

Likely causes and diagnostic signals

Symptom	Likely cause	Diagnostic signal
nvidia-smi works, but TF reports no CUDA driver	Driver/runtime mismatch	tf.sysconfig.get_build_info() shows a CUDA version the installed driver cannot satisfy
Import succeeds, no GPU listed	CPU-only TensorFlow build	Build info lacks CUDA/cuDNN entries
Errors naming libcudart, libcublas, or libcudnn	Missing shared libraries	Import or execution fails with .so load errors
GPU visible, then out-of-memory (OOM) errors	Allocator pressure	OOM only after operations begin; device list is non-empty
No GPU despite working nvidia-smi	Visibility environment variables	CUDA_VISIBLE_DEVICES=-1 or empty, or NVIDIA_VISIBLE_DEVICES misconfigured

Ordered checks

1. Confirm the host or container can see the GPU. Run nvidia-smi and nvidia-smi -q in the same shell where you launch Python. If this fails, the problem is driver installation or container GPU access, not TensorFlow.
2. Inspect the TensorFlow build in the failing environment. Run the diagnostic one-liner below. Compare the printed CUDA and cuDNN versions with the driver reported by nvidia-smi.
3. Check visibility overrides. Run env | grep -E 'CUDA_VISIBLE_DEVICES|NVIDIA_VISIBLE_DEVICES'. A value of -1 or an empty string hides all GPUs from CUDA processes.
4. Compare compiled versions with installed versions. Note the CUDA and cuDNN versions from tf.sysconfig.get_build_info() and verify they satisfy your driver and platform. Exact supported combinations vary by TensorFlow release; TF 2.11+ changed GPU packaging on Linux, so a combination that worked for one minor version may not work for another.
5. Rule out conflicting environments. Check whether multiple Python virtual environments, Conda environments, or system CUDA toolkits are active. A system CUDA toolkit installed over a TensorFlow wheel can shadow the libraries TF expects.
6. Run a minimal GPU operation. Use the explicit placement and memory-growth snippet below to confirm whether the device is truly usable.

Diagnostic one-liner

Run this in the exact Python environment that fails (after activating your virtual environment or entering your container shell). It requires only standard user permissions and is read-only.

python -c "import tensorflow as tf; print(tf.__version__); print(tf.config.list_physical_devices('GPU')); print(tf.sysconfig.get_build_info())"

Expected check: If the GPU list is empty but nvidia-smi works, proceed to environment variables and build-info comparison. If the build info contains no CUDA entries, you are running a CPU-only build.

Risk: None; the command does not modify state.

Minimal GPU operation with memory growth

Run this in a Python script or REPL in the same failing environment.

import tensorflow as tf

gpus = tf.config.list_physical_devices('GPU')
if gpus:
    for gpu in gpus:
        tf.config.experimental.set_memory_growth(gpu, True)
    result = tf.reduce_sum(tf.random.normal([512, 512]))
    print(result.device)
else:
    print("No GPU visible to TensorFlow")

Expected check: The printed device string should contain GPU. If it prints CPU or the GPU list is empty, the runtime cannot see the accelerator.

Risk: Minimal. If the GPU is visible but memory-constrained, reduce the tensor dimensions rather than disabling growth.

Fixes tied to findings

• nvidia-smi fails: Fix the host NVIDIA driver or container GPU access (for example, ensure the container was started with GPU support). This is outside TensorFlow.
• TensorFlow build lacks CUDA: Install the matching GPU-enabled TensorFlow package for your platform, or intentionally continue on CPU if that matches your deployment target.
• Missing shared libraries: Prefer the official TensorFlow GPU container image, or align the CUDA runtime and cuDNN versions with the TF build. Do not install a system CUDA toolkit over a TensorFlow wheel; that creates library conflicts.
• Environment variables hide the GPU: Correct or unset CUDA_VISIBLE_DEVICES and NVIDIA_VISIBLE_DEVICES in your shell, systemd unit, or container definition. Do not globally export these before diagnosing; it can mask the real failure.
• Allocator pressure (OOM): Keep set_memory_growth enabled, lower batch size, or use mixed precision and activation checkpointing to reduce memory footprint.
• Conflicting installs: Recreate a clean virtual environment and install only the GPU-enabled TensorFlow package plus your application dependencies.

Verifying the fix

After applying a change, re-run the diagnostic one-liner and the minimal GPU operation. Confirm that nvidia-smi shows your Python process using GPU memory while the operation runs. For intermittent failures, inspect kernel logs with dmesg or journalctl -k for NVIDIA Xid errors, ECC faults, or driver resets.

Treat first-run CUDA kernel compilation or ptxas messages as normal unless they end in fatal errors; confirm with repeated runs.

When to escalate

Escalate to platform or ML-infrastructure teams when:

• nvidia-smi works, but a known-good official TensorFlow GPU container still fails.
• Only some GPUs or distributed ranks fail in multi-GPU jobs.
• The issue reproduces after aligning driver and runtime versions in a clean image.
• Kernel logs show NVIDIA Xid errors, ECC faults, or driver resets.

Attach the TensorFlow version and build info, nvidia-smi output, container image tag, the exact launch command, and a minimal reproduction script.

Limitations

Exact supported CUDA, cuDNN, and driver combinations vary by TensorFlow release and platform. This guide does not enumerate version matrices; verify current compatibility against the official TensorFlow documentation for your specific version before changing system libraries.