Let Spark Coalesce Shuffle Partitions at Runtime with AQE
Adaptive Query Execution in Spark 3.x can resize shuffle partitions and handle skewed joins automatically, but it changes runtime plans and output file counts.
Loading…
ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Adaptive Query Execution in Spark 3.x can resize shuffle partitions and handle skewed joins automatically, but it changes runtime plans and output file counts.
AQE rewrites Spark physical plans at runtime using actual statistics, often reducing shuffle data by ~30%. This post explains the mechanism, a concrete example, trade-offs, and verification steps.
Static planning guesses at shuffle partition counts and join strategies. AQE re-decides after the shuffle, using real partition sizes — here's how to configure and verify it.