Spark job throws ClassNotFoundException for custom class when KryoSerializer enabled
0 reputation · 02 Dec 2025, 03:08 UTC
Goal: Ensure that a user‑defined class used in a Spark transformation is serializable when the application runs on a cluster with spark.serializer set to KryoSerializer.
In local mode the job completes because Spark falls back to the JVM’s default Java serializer for driver‑side objects, masking the missing Kryo registration. When submitted to a cluster the same code fails with a ClassNotFoundException for the custom class, indicating that Kryo cannot locate the class without explicit registration. The uncertainty lies in whether to register the class via spark.kryo.classesToRegister, to set spark.kryo.registrationRequired=false to suppress the check, or to rely on alternative serializers, considering version‑specific enforcement and potential performance impacts.
Should the custom class be added to spark.kryo.classesToRegister? Is disabling registrationRequired acceptable for production workloads? What performance trade‑offs arise from each option?
