Cannot load a saved Spark model in pyspark: "java.lang.NoSuchMethodException"

When I run the following Python program

from import LinearSVC
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Sparkmodel").getOrCreate()
data ="libsvm").load("/usr/local/spark/data/mllib/sample_libsvm_data.txt")
model = LinearSVC().fit(data)"mymodel")

the load fails with a "java.lang.NoSuchMethodException".

/anaconda3/envs/scratch/bin/python /Users/billmcn/src/toy/sparkmodel/sparkmodel/
Using Spark's default log4j profile: org/apache/spark/
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/11/12 13:23:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/12 13:23:06 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/11/12 13:23:06 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
17/11/12 13:23:17 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
17/11/12 13:23:17 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
Traceback (most recent call last):
  File "/Users/billmcn/src/toy/sparkmodel/sparkmodel/", line 9, in <module>
  File "/anaconda3/envs/scratch/lib/python3.6/site-packages/pyspark/ml/", line 257, in load
  File "/anaconda3/envs/scratch/lib/python3.6/site-packages/pyspark/ml/", line 197, in load
    java_obj = self._jread.load(path)
  File "/anaconda3/envs/scratch/lib/python3.6/site-packages/py4j/", line 1160, in __call__
    answer, self.gateway_client, self.target_id,
  File "/anaconda3/envs/scratch/lib/python3.6/site-packages/pyspark/sql/", line 63, in deco
    return f(*a, **kw)
  File "/anaconda3/envs/scratch/lib/python3.6/site-packages/py4j/", line 320, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o64.load.
: java.lang.NoSuchMethodException:<init>(java.lang.String)
    at java.lang.Class.getConstructor0(
    at java.lang.Class.getConstructor(
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(
    at java.lang.reflect.Method.invoke(
    at py4j.reflection.MethodInvoker.invoke(
    at py4j.reflection.ReflectionEngine.invoke(
    at py4j.Gateway.invoke(
    at py4j.commands.AbstractCommand.invokeMethod(
    at py4j.commands.CallCommand.execute(

Process finished with exit code 1

The "mymodel" directory is created and its contents appear to be valid.

I am running Spark 2.2.0 and pyspark 2.2.0. I have the following mllib jars in my installation.

> ll /usr/local/spark.versions/spark-2.2.0-bin-hadoop2.7/jars/spark-mllib*
-rw-r--r--@ 1 billmcn  admin  6501535 Jun 30 18:09 /usr/local/spark.versions/spark-2.2.0-bin-hadoop2.7/jars/spark-mllib_2.11-2.2.0.jar
-rw-r--r--@ 1 billmcn  admin   182887 Jun 30 18:09 /usr/local/spark.versions/spark-2.2.0-bin-hadoop2.7/jars/spark-mllib-local_2.11-2.2.0.jar

What am I doing wrong?

1 answer

  • answered 2017-11-12 19:49 user8929335

    This looks like a version mismatch. The most likely scenario is:

    • Your Python (PySpark) installation uses Spark 2.2
    • While JVM jars have been compiled with earlier Spark version, which didn't include LinearSVCModel.