Adaptive Query Execution (AQE) and Nested UDF Cost Estimation
Apache Spark's Adaptive Query Execution (AQE) optimizes query plans at runtime by utilizing statistics from completed shuffle stages. This allows the engine to dynamically coalesce shuffle partitions and switch join strategies, such as converting a SortMergeJoin to a BroadcastHashJoin when actual relation sizes fall below the threshold. However, the integrat