What happens when using LIMIT/OFFSET in Spark without an explicit ORDER BY clause?
0 reputation · 24 Mar 2021, 00:56 UTC
Goal: determine if Spark applies a global sort when LIMIT/OFFSET used without ORDER BY and what performance implications arise.
Constraints: Spark does not guarantee partition order; without ORDER BY, LIMIT/OFFSET may trigger a shuffle to sort the entire dataset before returning subset, leading to nondeterministic results and potential memory pressure on skewed data.
Does Spark implicitly sort the whole dataset when LIMIT/OFFSET lacks ORDER BY? What shuffle volume and stage duration can be observed in the Spark UI? How does data skew affect the cost of this implicit sort?