Spark 4.0 drops Scala 2.12 support, breaking Scala binary compatibility

Compatibility Policy and Requirements

Spark 4.0 officially drops support for Scala 2.12. The binary compatibility boundary is broken between Spark 3.5 (which supported both 2.12 and 2.13) and Spark 4.0 (which requires Scala 2.13). Consequently, any application or library compiled against Spark 3.5 using Scala 2.12 is binary incompatible with the Spark 4.0 runtime.

Additionally, Spark 4.0 mandates Java 17 at runtime, removing support for Java 8 and 11. This creates a dual-migration requirement for legacy toolchains.

Bridging and Recompilation

Spark 4.0 does not provide bridging artifacts for Scala 2.12. Recompilation to Scala 2.13 is mandatory for all Scala-based codebases. Attempting to run Scala 2.12 binaries on a Spark 4.0 cluster will result in NoSuchMethodError or ClassNotFoundException due to changes in the Scala standard library and Spark's internal API signatures.

Mixed-Version Cluster Planning

When managing clusters where client libraries and the server differ, consider the following architectural constraints:

• Standard Spark Clients: These must match the server's Scala and Spark versions. A 3.x client cannot communicate with a 4.0 server via traditional driver-executor mechanisms.
• Spark Connect: Because Spark Connect decouples the client from the server using a gRPC-based protocol, clients built for Spark 3.x may maintain connectivity to a Spark 4.0 server, provided the API surface used is stable. However, this should be verified against the specific Spark 4.0 release notes for any breaking changes in the Connect protocol.

Migration Steps

1. Update Toolchain: Upgrade the JDK to version 17.
2. Update Build Configuration: Change the scala-version in build.sbt or the Scala version property in pom.xml to 2.13.x.
3. Dependency Audit: Identify third-party libraries compiled only for Scala 2.12; these must be updated to 2.13 versions or replaced.
4. Recompile and Test: Build the project and execute a smoke test to verify the initialization of the SparkSession.

Verification Command:
spark-submit --version
Ensure the output confirms the Spark 4.0 version and the associated Scala 2.13 runtime.

Diagnostic Detail Needed: Are you utilizing any third-party Scala libraries that lack a 2.13 distribution? This would necessitate a wrapper or a library replacement before the migration can proceed.