ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Developer tools · #apachespark
Spark's static optimizer often picks expensive joins and creates tiny shuffle partitions when table statistics are stale. Adaptive Query Execution re-optimizes the physical plan at runtime using actual cardinalities, coalescing partitions and switching join strategies on the fly.
Learn how to diagnose and resolve Java heap space OutOfMemoryErrors during Apache Spark shuffles, including identifying data skew and optimizing shuffle partitions.
Spark's Adaptive Query Execution re-optimizes plans at runtime, but only if you understand its limits. Learn how to fix skewed join stragglers, configure partition coalescing, and verify the final plan in the Spark UI.
Speed up joins between a large dataset and a small lookup table in Spark by using broadcast joins. Learn how to set thresholds, apply hints, verify the plan, and recover from failures.
Broadcast joins eliminate shuffling for small tables in Spark. This post explains how to enable them, shows a concrete example, and discusses memory limits and performance trade‑offs for developers.
A decision guide for Spark 3.x: when to rely on Adaptive Query Execution for shuffle partition sizing, skew handling, and join conversion versus manually tuning spark.sql.shuffle.partitions, with concrete validation steps.
AQE rewrites Spark physical plans at runtime using actual statistics, often reducing shuffle data by ~30%. This post explains the mechanism, a concrete example, trade-offs, and verification steps.
Spark's broadcast join eliminates shuffle when one side of a join is small — but the optimizer doesn't always choose it. Here's how to force it, verify it, and avoid the OOM trap.
Learn how Spark’s broadcast join can eliminate costly shuffles, when it applies automatically, and how to force it or avoid pitfalls with a concrete example.