MEMORY_ONLY versus MEMORY_AND_DISK: recomputation cost or spill latency under tight executor heap
For a Spark 3.x batch job that reuses the same DataFrame across several stages, the choice between MEMORY_ONLY and MEMORY_AND_DISK becomes a trade-off between recomputation overhead and spill latency. With MEMORY_ONLY , Spark stores deserialized partitions in heap and evicts them entirely when memory is tight, forcing the lineage to be recomputed on the next