MEMORY_ONLY versus MEMORY_AND_DISK: recomputation cost or spill latency under tight executor heap

Question

For a Spark 3.x batch job that reuses the same DataFrame across several stages, the choice between MEMORY_ONLY and MEMORY_AND_DISK becomes a trade-off between recomputation overhead and spill latency. With MEMORY_ONLY, Spark stores deserialized partitions in heap and evicts them entirely when memory is tight, forcing the lineage to be recomputed on the next access. MEMORY_AND_DISK avoids that recomputation by writing evicted partitions to local disk, but subsequent reads incur disk I/O.
The uncertainty is which cost dominates for a given workload. Recomputing a partition may be cheaper than reading it from local disk if the transformation is simple, but more expensive if the lineage is deep. Executor heap size, partition granularity, and deserialized object overhead all shift the balance. The Spark UI Storage tab can reveal memory usage and spill after the fact, but predicting the eviction threshold before execution remains difficult.
Which persistence level should be preferred when executor heap is constrained and the dataset is reused multiple times? How can the recomputation cost be estimated relative to disk spill latency for a specific job? Does the presence of other cached datasets on the same executors change the eviction behavior enough to alter the decision?

Tasadduq Burney · Answer

Direct answer
For a Spark 3.x batch job that reuses the same DataFrame across several stages with a tightly constrained executor heap, prefer MEMORY_AND_DISK when reuse is more than a few times and the lineage is deep or reads remote input. Prefer MEMORY_ONLY only when reuse is one or two times and the transformation is cheap enough that recomputation beats a local disk read. Under extreme heap pressure, neither deserialized level is ideal; MEMORY_AND_DISK_SER or MEMORY_ONLY_SER usually reduces heap footprint more than the two non-serialized options.

Confirmed facts versus likely explanation
Confirmed facts

Spark 3.x uses unified memory, so storage and execution share one pool; spark.memory.storageFraction influences the cache share.
Other cached datasets, broadcasts, and shuffle execution compete for the same storage memory and shrink space for this DataFrame.
MEMORY_ONLY keeps deserialized partitions in heap and drops them when the storage store is full; later access recomputes the full lineage.
MEMORY_AND_DISK retains hot blocks in heap and writes evicted blocks to local executor disk; repeated reads avoid recomputation but pay disk I/O on misses.
Spark's BlockManager evicts blocks when the memory store is full; there is no universal pre-execution eviction threshold.
Large deserialized heaps can make garbage-collection time dominate both recomputation and disk latency.

Likely explanation (verify for your job)

If lineage is shallow and input is local, recompute cost may be lower than disk spill latency.
If lineage is deep or reads remote storage, disk spill usually wins despite I/O cost.
Additional cached datasets increase eviction frequency and can cause memory-to-disk thrashing, weakening MEMORY_AND_DISK's advantage.

Steps for this case

Run a representative subset once with MEMORY_ONLY and once with MEMORY_AND_DISK; compare end-to-end time, executor GC time, and Spark UI Storage tab residency and eviction.
Time one lineage execution for a single partition without caching to establish the recompute baseline.
Time a cached partition read after eviction, or compare MEMORY_AND_DISK task durations on cache hits versus misses.
If recompute time per partition is lower than expected disk-read misses multiplied by local disk read latency, choose MEMORY_ONLY; otherwise choose MEMORY_AND_DISK.
Check the aggregate cached size of all datasets on the same executors before finalizing; shared cache pressure can flip the decision.
If heap remains tight, retest with MEMORY_AND_DISK_SER to reduce deserialized object overhead.

Missing diagnostic that can change the recommendation
Provide measured recompute time per partition versus measured local disk read time for an evicted block, together with the expected reuse count. Without those numbers, the choice remains a guess.

Verification
Open the Spark UI Storage tab at http://DRIVER_HOST:4040/storage/ after the run and confirm block storage level, memory used, disk used, and eviction counts. Review executor logs for block evictions and task deserialization time.

--conf spark.memory.storageFraction=0.6
--conf spark.executor.memory=4g

MEMORY_ONLY versus MEMORY_AND_DISK: recomputation cost or spill latency under tight executor heap

1 answer

Direct answer

Confirmed facts versus likely explanation

Confirmed facts

Likely explanation (verify for your job)

Steps for this case

Missing diagnostic that can change the recommendation

Verification

Your perspective belongs here.

0 question comments