Spark 4.0 drops Scala 2.12 support, breaking Scala binary compatibility

Spark 4.0 changes the Scala binary compatibility boundary compared with Spark 3.5. Spark 3.5 ships Scala 2.12 and 2.13 builds, while Spark 4.0 is documented as Scala 2.13 only and requires Java 17 at runtime.

The goal is to determine the impact on existing codebases compiled against Spark 3.5 with Scala 2.12 and Java 8/11 toolchains. The constraints include the runtime Java requirement increase, the loss of Scala 2.12 artifacts, and the need to understand whether Spark Connect clients built for Spark 3.x remain usable against a Spark 4.0 server.

What is the official compatibility policy for Scala binary versions across the Spark 3.5 to 4.0 boundary? Does Spark 4.0 provide any bridging artifacts for Scala 2.12 libraries, or is recompilation to Scala 2.13 mandatory? How should mixed-version clusters be planned when client libraries and the server differ by minor version?