Spark job throws ClassNotFoundException for custom class when KryoSerializer enabled

Answer the Question

The ClassNotFoundException is thrown because the executor’s classloader cannot locate the user‑defined class when KryoSerializer is in use. The fix is to make the class available on the executors and, in most production workloads, register it with Kryo.

Why the Exception Happens

When spark.serializer is set to KryoSerializer, Spark serializes all objects that cross the driver‑executor boundary with Kryo. Kryo requires that every class used in a transformation be either (a) registered in spark.kryo.classesToRegister or (b) discoverable on the executor classpath. If the class is missing, Kryo cannot instantiate a ClassResolver for it and throws ClassNotFoundException. In local mode the JVM’s default serializer is used for driver‑side objects, so the missing registration is hidden.

Recommended Fixes

• Ship the JAR: Ensure the JAR that contains the custom class is added to the executor classpath. Use --jars /path/to/your.jar or publish the JAR to a Maven repository and reference it with --packages.
• Register the class: Add the fully‑qualified class name to spark.kryo.classesToRegister (via SparkConf, spark-defaults.conf, or the command line). This gives Kryo a numeric ID for the class, eliminating the need for reflection and reducing payload size.
• Do not rely on registrationRequired=false in production. Turning off the check removes a safety net that would surface accidental use of non‑serializable classes and forces Kryo to fall back to slower, larger reflection‑based serialization.

Performance Trade‑offs

Option	Serialization Size	Speed	Safety
Registered class	Small (numeric ID)	Fast	High – errors surface early
Unregistered with registrationRequired=false	Large (class name + data)	Slow	Low – may hide bugs

Verification Steps

1. Run spark-submit --jars path/to/your.jar \ --conf spark.kryo.classesToRegister=com.example.MyClass \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ yourApp.jar.
2. Check executor logs for a line similar to Class com.example.MyClass loaded by executor classloader.
3. Use Spark UI’s Storage tab to confirm that the RDD partitions have a smaller serialized size after registration.
4. If you set spark.kryo.registrationRequired=false, compare the execution time and executor memory usage against the registered case to quantify the penalty.

Missing Diagnostic Detail

Do you ship the JAR containing com.example.MyClass to the cluster executors (via --jars or a repository)? If not, that is the most likely cause of the exception.