Stopping the Shuffle: Optimizing Spark Joins with Broadcast Hash Joins
Stop wasting resources on unnecessary shuffles. Learn how to use Broadcast Hash Joins in Apache Spark to optimize joins between large and small datasets.
Loading…
ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Stop wasting resources on unnecessary shuffles. Learn how to use Broadcast Hash Joins in Apache Spark to optimize joins between large and small datasets.
Spark 3.x Adaptive Query Execution reoptimizes shuffle partitioning, join strategy and skew handling at runtime using actual shuffle statistics instead of static estimates.
Enable Spark 3.x Adaptive Query Execution to coalesce shuffle partitions and prune joins at runtime, then verify with explain and the Spark UI before rolling it out.
Spark's static optimizer often picks expensive joins and creates tiny shuffle partitions when table statistics are stale. Adaptive Query Execution re-optimizes the physical plan at runtime using actual cardinalities, coalescing partitions and switching join strategies on the fly.
Spark's broadcast join eliminates shuffle when one side of a join is small — but the optimizer doesn't always choose it. Here's how to force it, verify it, and avoid the OOM trap.
A decision guide for Spark broadcast joins: when the small side is small enough, how to tune autoBroadcastJoinThreshold safely, and how to verify the plan actually changed.
AQE can switch join strategies and split skewed partitions, but only at shuffle boundaries. A worked configuration, how to verify the plan changed, and where it stops helping.
Architecture note on enabling and operating Apache Spark Adaptive Query Execution: requirements, minimal config, trust boundaries, checks, failure modes, and redesign triggers.
Static planning guesses at shuffle partition counts and join strategies. AQE re-decides after the shuffle, using real partition sizes — here's how to configure and verify it.