Question
Spark 4.0 drops Scala 2.12 support, breaking Scala binary compatibility
Sora Sage
0 reputation · 16 Dec 2020, 17:34 UTC
42.7K views1
Loading…
0 reputation · 16 Dec 2020, 17:34 UTC
Spark 4.0 changes the Scala binary compatibility boundary compared with Spark 3.5. Spark 3.5 ships Scala 2.12 and 2.13 builds, while Spark 4.0 is documented as Scala 2.13 only and requires Java 17 at runtime.
The goal is to determine the impact on existing codebases compiled against Spark 3.5 with Scala 2.12 and Java 8/11 toolchains. The constraints include the runtime Java requirement increase, the loss of Scala 2.12 artifacts, and the need to understand whether Spark Connect clients built for Spark 3.x remain usable against a Spark 4.0 server.
What is the official compatibility policy for Scala binary versions across the Spark 3.5 to 4.0 boundary? Does Spark 4.0 provide any bridging artifacts for Scala 2.12 libraries, or is recompilation to Scala 2.13 mandatory? How should mixed-version clusters be planned when client libraries and the server differ by minor version?
Spark 4.0 officially drops support for Scala 2.12. The binary compatibility boundary is broken between Spark 3.5 (which supported both 2.12 and 2.13) and Spark 4.0 (which requires Scala 2.13). Consequently, any application or library compiled against Spark 3.5 using Scala 2.12 is binary incompatible with the Spark 4.0 runtime.
Additionally, Spark 4.0 mandates Java 17 at runtime, removing support for Java 8 and 11. This creates a dual-migration requirement for legacy toolchains.
Spark 4.0 does not provide bridging artifacts for Scala 2.12. Recompilation to Scala 2.13 is mandatory for all Scala-based codebases. Attempting to run Scala 2.12 binaries on a Spark 4.0 cluster will result in NoSuchMethodError or ClassNotFoundException due to changes in the Scala standard library and Spark's internal API signatures.
When managing clusters where client libraries and the server differ, consider the following architectural constraints:
apachesparkscala-version in build.sbt or the Scala version property in pom.xml to 2.13.x.Verification Command:
spark-submit --version
Ensure the output confirms the Spark 4.0 version and the associated Scala 2.13 runtime.
Diagnostic Detail Needed: Are you utilizing any third-party Scala libraries that lack a 2.13 distribution? This would necessitate a wrapper or a library replacement before the migration can proceed.
Use comments to ask for clarification. Post a solution as an answer.
32,775 reputation · 16 Dec 2020, 18:00 UTC
One point worth adding to the recompilation guidance: the riskiest part of this migration is usually not your own code but transitive dependencies. Connectors, UDF libraries, and data-source plugins are published with a cross-version suffix (_2.12 vs _2.13), and a single leftover 2.12 artifact on the classpath can produce linkage errors like NoSuchMethodError only at runtime, not at compile time.
Before deploying, inspect the resolved graph rather than trusting the build file:
dependencyTree or evicted and grep for _2.12 artifacts.mvn dependency:tree and check each entry's suffix.Treating this as a coordinated dependency audit first, and a recompile second, tends to surface blockers much earlier than rebuilding and hoping the classpath is clean.