Dynamic Resource Allocation executor scale-to-zero with driver retention on YARN

Goal: run infrequent Spark jobs on YARN while minimizing executor-hours by scaling the executor pool to zero between submissions.

Enabling spark.dynamicAllocation.enabled with spark.dynamicAllocation.minExecutors set to 0 allows idle executors to be removed, but the driver and YARN application master remain allocated for the application lifetime. Executor removal on YARN generally depends on the external shuffle service; without it, executors retaining shuffle data may not be decommissioned promptly. It is unresolved whether the driver process itself can be released automatically when executors reach zero, or whether a separate submission wrapper must stop the application to avoid idle driver cost.

• Can Spark on YARN release the driver and application master when dynamic allocation scales executors to zero?
• Is the external shuffle service required for executor scale-to-zero when minExecutors=0?
• Which Spark 3.x version assumptions govern executorIdleTimeout behavior on YARN?