Cache vs Prefetch: Choosing tf.data Dataset.cache() or Dataset.prefetch() for Limited Host RAM

Goal

Determine the most appropriate tf.data strategy for a training pipeline that must reuse the same data across multiple epochs while staying within a constrained host RAM budget.

Constraints & Uncertainty

• Host RAM is limited to 4 GB, but the dataset contains many large images, so in‑memory caching may be costly.
• Each epoch repeats the same examples, but the pipeline includes a tf.image.decode_jpeg map and a tf.data.shuffle step.
• There is uncertainty whether the observed post‑warm‑up memory plateau is a genuine leak or merely TensorFlow allocator caching.
• It is unclear how placing cache() before or after shuffle() affects epoch‑to‑epoch example ordering and memory consumption.

Unresolved Decision

The trade‑off revolves around whether to use Dataset.cache() to materialize data for reuse or Dataset.prefetch() to overlap I/O and CPU work. cache() can reduce repeated decoding but may increase memory usage, whereas prefetch() keeps memory bounded but does not eliminate per‑epoch recomputation.

Specific Questions

1. When cache() is applied before shuffle, does TensorFlow freeze the shuffle order across epochs, effectively making each epoch identical?
2. What are the comparative memory footprints of cache() placed before versus after the decoding map, given the same dataset cardinality?
3. Which diagnostics best distinguish a true memory leak from allocator caching when the process RSS remains high after training starts?