Dynamic Resource Allocation executor scale-to-zero with driver retention on YARN

Question

Goal: run infrequent Spark jobs on YARN while minimizing executor-hours by scaling the executor pool to zero between submissions.
Enabling spark.dynamicAllocation.enabled with spark.dynamicAllocation.minExecutors set to 0 allows idle executors to be removed, but the driver and YARN application master remain allocated for the application lifetime. Executor removal on YARN generally depends on the external shuffle service; without it, executors retaining shuffle data may not be decommissioned promptly. It is unresolved whether the driver process itself can be released automatically when executors reach zero, or whether a separate submission wrapper must stop the application to avoid idle driver cost.

Can Spark on YARN release the driver and application master when dynamic allocation scales executors to zero?
Is the external shuffle service required for executor scale-to-zero when minExecutors=0?
Which Spark 3.x version assumptions govern executorIdleTimeout behavior on YARN?

Muntasir · Answer

Executor Scale-to-Zero Behavior on YARN
Dynamic resource allocation with spark.dynamicAllocation.minExecutors=0 successfully scales executor containers down to zero. However, this action does not automatically release the Spark driver process or the YARN ApplicationMaster (AM) container.

Confirmed Facts

Executor containers can be released down to zero when idle, contingent on shuffle service configuration.
The driver and AM remain allocated by YARN for the entire application lifecycle, regardless of executor count.
To release driver/AM resources, the Spark application must terminate cleanly (e.g., via SparkContext.stop()) or the YARN application must be explicitly killed by an external wrapper.

Necessary Steps for Zero Executor Cost

Set spark.dynamicAllocation.enabled to true and spark.dynamicAllocation.minExecutors to 0.
Ensure spark.shuffle.service.enabled is true and the external shuffle service is running on YARN NodeManagers for prompt executor cleanup.
Configure a short spark.dynamicAllocation.executorIdleTimeout for testing purposes.
Use a job wrapper or ensure the Spark job completes naturally to trigger driver shutdown, allowing YARN to release the AM container.

Uncertainty and Next Diagnostic Step
The precise timing of executor removal when shuffle data is present depends heavily on the external shuffle service status. To confirm if the driver/AM *can* be released automatically upon executor zeroing, please verify if the YARN ApplicationMaster container status transitions to 'FINISHED' or 'KILLED' immediately after the last executor is removed, or if it persists until the job script explicitly exits.

Dynamic Resource Allocation executor scale-to-zero with driver retention on YARN

1 answer

Executor Scale-to-Zero Behavior on YARN

Confirmed Facts

Necessary Steps for Zero Executor Cost

Uncertainty and Next Diagnostic Step

Your perspective belongs here.

0 question comments