ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Developer tools · #apachespark
Decide between Spark Structured Streaming and the legacy DStream API for new real-time pipelines on Spark 3.x.
Pick a watermark delay by measuring event lateness, then verify with state-store metrics and console output.
Adaptive Query Execution in Spark 3.x can resize shuffle partitions and handle skewed joins automatically, but it changes runtime plans and output file counts.
Decision guide for choosing between broadcast hash join and sort-merge join in Apache Spark 3.x, with constraints, trade-offs and a validation workflow using EXPLAIN and Spark UI.
Broadcast Hash Join cuts join latency by eliminating shuffles when one side is small. Learn when to enable it, how to force it, and the trade‑offs to watch for in real‑world Spark jobs.
Spark broadcast joins avoid shuffling the large side of a join when one dataset fits in memory. This post walks through the configuration, a concrete example, and how to verify the improvement.
Stop wasting resources on unnecessary shuffles. Learn how to use Broadcast Hash Joins in Apache Spark to optimize joins between large and small datasets.
Spark 3.x Adaptive Query Execution reoptimizes shuffle partitioning, join strategy and skew handling at runtime using actual shuffle statistics instead of static estimates.
Enable Spark 3.x Adaptive Query Execution to coalesce shuffle partitions and prune joins at runtime, then verify with explain and the Spark UI before rolling it out.