Spark AQE vs Manual Shuffle Partition Tuning: A Decision Guide for Large Joins
A decision guide for Spark 3.x: when to rely on Adaptive Query Execution for shuffle partition sizing, skew handling, and join conversion versus manually tuning spark.sql.shuffle.partitions, with concrete validation steps.