Choosing Between Map-style and Iterable Datasets in PyTorch

The Data Loading Bottleneck

Selecting the wrong PyTorch dataset architecture often leads to two critical failures: memory exhaustion (OOM) when loading massive files or severe I/O latency when randomly accessing remote cloud storage. The core decision rests on whether your data can be indexed randomly (Map-style) or must be processed as a continuous stream (Iterable-style).

Decision Matrix: Map-style vs. IterableDataset

Constraint	Map-style Dataset	IterableDataset
Data Access	Random access via index	Sequential stream
Required Methods	__len__ and __getitem__	__iter__
Shuffling	Handled by DataLoader sampler	Manual (Shuffle Buffer)
Ideal Source	Local SSD, RAM, Indexed DB	Network sockets, Huge CSVs, S3 streams
Worker Logic	Automatic splitting	Manual worker-splitting required

Trade-offs and Engineering Risks

Map-style Datasets are the standard for most projects. By implementing __getitem__, you allow the DataLoader to request any sample at any time. This enables the built-in shuffle=True parameter to work efficiently. However, if your data resides in a remote cloud bucket, random access triggers a new HTTP request for every sample, creating a massive I/O bottleneck.

IterableDatasets solve the I/O problem by reading data sequentially. This is essential for datasets that exceed local disk capacity. The trade-off is complexity: you lose the DataLoader's automatic shuffling and data splitting. If you use num_workers > 0 without custom logic, every worker process will read the exact same stream, duplicating your data and biasing your gradients.

Implementing a Safe IterableDataset

To use an IterableDataset with multi-process loading (PyTorch 2.0+), you must explicitly partition the data based on the worker ID. This prevents the duplication risk mentioned above.

import torch
from torch.utils.data import IterableDataset, DataLoader
import math

class StreamingDataset(IterableDataset):
    def __init__(self, data_source):
        self.data_source = data_source

    def __iter__(self):
        # Get worker information to avoid duplicate data
        worker_info = torch.utils.data.get_worker_info()
        
        if worker_info is None:
            # Single-process loading
            iter_start = 0
            iter_end = len(self.data_source)
        else:
            # Multi-process loading: split the workload
            per_worker = int(math.ceil(len(self.data_source) / float(worker_info.num_workers)))
            worker_id = worker_info.id
            iter_start = worker_id * per_worker
            iter_end = min(iter_start + per_worker, len(self.data_source))

        return iter(self.data_source[iter_start:iter_end])

# Example Usage
raw_data = list(range(100))
dataset = StreamingDataset(raw_data)
# Run this on a machine with multiple CPU cores
loader = DataLoader(dataset, batch_size=10, num_workers=2)

for batch in loader:
    print(batch)

Verification and Diagnostics

To verify your implementation is correct, perform these two checks:

• Duplication Check: Set num_workers=2. Collect all samples from one epoch into a list. If the list length is exactly the size of your dataset, the splitting logic is working. If the length is double, you have a duplication bug.
• Memory Profiling: When switching from Map-style to Iterable, monitor the Resident Set Size (RSS) of your worker processes. You should see a flat memory profile regardless of dataset size, as only the current buffer is held in RAM.

Limitations

IterableDatasets do not support the sampler argument in DataLoader. If you require global shuffling across a multi-terabyte dataset, you must implement a shuffle buffer: load a fixed number of elements into a queue, randomly sample from that queue, and refill it as elements are consumed.