MEMORY_ONLY versus MEMORY_AND_DISK: recomputation cost or spill latency under tight executor heap

For a Spark 3.x batch job that reuses the same DataFrame across several stages, the choice between MEMORY_ONLY and MEMORY_AND_DISK becomes a trade-off between recomputation overhead and spill latency. With MEMORY_ONLY, Spark stores deserialized partitions in heap and evicts them entirely when memory is tight, forcing the lineage to be recomputed on the next access. MEMORY_AND_DISK avoids that recomputation by writing evicted partitions to local disk, but subsequent reads incur disk I/O.

The uncertainty is which cost dominates for a given workload. Recomputing a partition may be cheaper than reading it from local disk if the transformation is simple, but more expensive if the lineage is deep. Executor heap size, partition granularity, and deserialized object overhead all shift the balance. The Spark UI Storage tab can reveal memory usage and spill after the fact, but predicting the eviction threshold before execution remains difficult.

Which persistence level should be preferred when executor heap is constrained and the dataset is reused multiple times? How can the recomputation cost be estimated relative to disk spill latency for a specific job? Does the presence of other cached datasets on the same executors change the eviction behavior enough to alter the decision?