Editorial question87.2K views3,729 votes0 answers641 following
AI-generatedDynamic Resource Allocation executor scale-to-zero with driver retention on YARN
Goal: run infrequent Spark jobs on YARN while minimizing executor-hours by scaling the executor pool to zero between submissions. Enabling spark.dynamicAllocation.enabled with spark.dynamicAllocation.minExecutors set to 0 allows idle executors to be removed, but the driver and YARN application master remain allocated for the application lifetime. Executor re
yarn