Spark Structured Streaming vs DStream: A Decision Guide
Decide between Spark Structured Streaming and the legacy DStream API for new real-time pipelines on Spark 3.x.
Loading…
ReadMeFeed / Community knowledge
#apache-spark
Real questions. Useful conversations. Find the people who know your stack.
Choosing a Watermark Delay for Spark Structured Streaming State
Pick a watermark delay by measuring event lateness, then verify with state-store metrics and console output.
Incremental Materialized View Refresh in Rollup: Engineering Trade-offs for Spark Analytics
Rollup's incremental materialized view refresh reduces recomputation overhead by up to 90% for Spark workloads, but requires careful configuration around partition keys, trigger intervals, and join complexity to avoid fallback to full refresh.
rollupLet Spark Coalesce Shuffle Partitions at Runtime with AQE
Adaptive Query Execution in Spark 3.x can resize shuffle partitions and handle skewed joins automatically, but it changes runtime plans and output file counts.
Choosing Broadcast Hash Join vs Sort-Merge Join in Apache Spark 3.x
Decision guide for choosing between broadcast hash join and sort-merge join in Apache Spark 3.x, with constraints, trade-offs and a validation workflow using EXPLAIN and Spark UI.
Stopping the Shuffle: Optimizing Spark Joins with Broadcast Hash Joins
Stop wasting resources on unnecessary shuffles. Learn how to use Broadcast Hash Joins in Apache Spark to optimize joins between large and small datasets.
Use Adaptive Query Execution to Reoptimize Spark 3.x Joins and Shuffle Partitions at Runtime
Spark 3.x Adaptive Query Execution reoptimizes shuffle partitioning, join strategy and skew handling at runtime using actual shuffle statistics instead of static estimates.
Tune Spark 3.x Batch ETL Shuffle Partitions with Adaptive Query Execution
Enable Spark 3.x Adaptive Query Execution to coalesce shuffle partitions and prune joins at runtime, then verify with explain and the Spark UI before rolling it out.
Stale Statistics and Bad Joins: How Spark's Adaptive Query Execution Fixes Plans Mid-Flight
Spark's static optimizer often picks expensive joins and creates tiny shuffle partitions when table statistics are stale. Adaptive Query Execution re-optimizes the physical plan at runtime using actual cardinalities, coalescing partitions and switching join strategies on the fly.