Dynamic Resource Allocation executor scale-to-zero with driver retention on YARN
Goal: run infrequent Spark jobs on YARN while minimizing executor-hours by scaling the executor pool to zero between submissions. Enabling spark.dynamicAllocation.enabled with spark.dynamicAllocation.minExecutors set to 0 allows idle executors to be removed, but the driver and YARN application master remain allocated for the application lifetime. Executor re
yarn