DataLoader timeout limits: worker processes persist after RuntimeError

Direct answer

A DataLoader timeout is a main-process wait limit only. When it fires, PyTorch raises RuntimeError in the main process but sends no cancellation signal to the worker. The stuck __getitem__ and its worker process continue running. There is no supported API to interrupt a single worker, replace only that worker, or reset the loader’s iteration state cleanly.

Treat the timeout as a hard failure for that DataLoader instance. Stop iteration, discard the loader, and create a new one (which spawns a fresh worker pool). Do not call __iter__ or next() again on the same instance.

Why the worker survives

The timeout argument is implemented as a deadline on the internal queue get() that the coordinator thread uses to fetch the next batch. It is not a worker lifecycle controller. When the deadline expires, the coordinator raises RuntimeError and abandons the queue read; the worker process remains alive, still executing the slow __getitem__ or blocked on I/O, locks, or GPU operations.

What happens if you keep using the same loader

Behavior is undefined and version-dependent. Observed outcomes include:

• The next next() hangs indefinitely waiting for a result that will never arrive.
• Duplicate work: the orphaned worker eventually pushes its batch into the queue, and a subsequent iteration consumes it out of order.
• Orphaned workers persist after the main process exits, holding file descriptors, CUDA contexts, or shared memory.
• persistent_workers=True makes stuck workers more visible across epochs but does not add per-worker cancellation.

Recommended recovery pattern

1. Catch RuntimeError (or the specific torch.utils.data._utils.worker.TimeoutError subclass if available in your version).
2. Break out of the training/validation loop immediately.
3. Explicitly delete the DataLoader reference and, if practical, call torch.utils.data._utils.worker._worker_loop cleanup helpers—though the only reliable path is to let the process exit or recreate the loader in a fresh process.
4. Instantiate a new DataLoader with the same configuration to get a clean worker pool.

One diagnostic detail that changes the recommendation

What multiprocessing start method are you using (fork, spawn, or forkserver) and what PyTorch version? Cleanup semantics (whether __del__ terminates workers, how queues are drained, and whether persistent_workers interacts with timeout) differ across these combinations. If you are on spawn or forkserver with a recent PyTorch (≥2.1), worker teardown on loader deletion is more deterministic, but the no-single-worker-replacement rule still holds.