Blog52K views4,229 votes0 replies560 following
Tune Spark 3.x Batch ETL Shuffle Partitions with Adaptive Query Execution
Enable Spark 3.x Adaptive Query Execution to coalesce shuffle partitions and prune joins at runtime, then verify with explain and the Spark UI before rolling it out.