Adaptive Query Execution (AQE) and Nested UDF Cost Estimation

Apache Spark's Adaptive Query Execution (AQE) optimizes query plans at runtime by utilizing statistics from completed shuffle stages. This allows the engine to dynamically coalesce shuffle partitions and switch join strategies, such as converting a SortMergeJoin to a BroadcastHashJoin when actual relation sizes fall below the threshold.

However, the integration of complex nested User Defined Functions (UDFs) introduces uncertainty into the optimizer's cost estimation. Because UDFs are treated as black boxes, the runtime statistics collected during the shuffle phase may not accurately reflect the computational overhead or the data distribution changes caused by nested logic.

Design Constraints

• AQE relies on runtime statistics from the shuffle phase to re-plan stages.
• Nested UDFs can obscure the actual cost and cardinality of the resulting dataset.
• Driver overhead increases when coordinating frequent re-planning of complex stages.

In Spark 3.x, how does the AQE optimizer handle cost estimation for stages where nested UDFs precede the shuffle? Does the framework provide a mechanism to hint the actual cost of these UDFs to prevent suboptimal join strategy selection?