Can Catalyst physical planning move shuffle stage boundaries in Spark 3.x?

Answer

Catalyst physical planning cannot directly remove or merge existing shuffle stage boundaries; stage boundaries are established by the logical plan’s shuffle operators. Only if the logical plan eliminates a shuffle (e.g., by choosing a broadcast join) can a stage be merged, which may appear as a boundary change.

Confirmed facts

• Stage boundaries are fixed after logical optimization; physical planning selects execution strategies within the defined stage.
• Catalyst can rewrite the logical plan (e.g., replace a shuffle hash join with a broadcast join) but does not create new shuffle stages.
• If a broadcast join is applicable and removes the need for a shuffle, the stage containing that shuffle may be merged with the preceding stage, effectively moving the boundary.
• Spark UI stage IDs and shuffle read/write metrics are the reliable indicators of a boundary change.

Steps to verify

1. Record the stage ID(s) and shuffle metrics for the query before physical planning.
2. Run spark.sql("explain") to view the physical plan and confirm whether a shuffle operator is present.
3. Check whether the join’s smaller side meets the broadcast join threshold (size‑based rule).

Missing diagnostic detail

What is the size of the smaller join operand relative to the broadcast join threshold? This determines whether Catalyst can eliminate the shuffle and thus merge stages.