Spark-Submit Error: Cannot load main class from JAR file

I am trying to spark-submit an application in Scala cluster mode.It was working fine in PySpark but while trying to run with Scala the above error is popping up. If I have to add SBT and Maven dependencies can you elaborate the procedure(I am not able to find in Google)

This is my code:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object MyFirst {
  def main(args: Array[String]) {
    // create Spark context with Spark configuration
    val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

    // get threshold
    val threshold = args(1).toInt

    // read in text file and split each document into words
    val tokenized = sc.textFile(args(0)).flatMap(_.split(" "))

    // count the occurrence of each word
    val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)

    // filter out words with fewer than threshold occurrences
    val filtered = wordCounts.filter(_._2 >= threshold)

    // count characters
    val charCounts = filtered.flatMap(_._1.toCharArray).map((_, 1)).reduceByKey(_ + _)

    System.out.println(charCounts.collect().mkString(", "))
  }
}

Here is my Build.sbt

name := "MyFirst"

scalaVersion := "2.10.3"

// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"

My Spark Submit was: spark-submit MyFirst --class MyFirst /home/ram/Downloads/sbt/src/target/scala-2.10/MyFirst_2.10-0.1.0-SNAPSHOT.jar

1 answer

  • answered 2018-11-09 05:06 Carlos David Peña

    I had this issue before.

    Try defining a package for your class, something like this:

    package com.testing
    
    import org.apache.spark.SparkContext
    import org.apache.spark.SparkContext._
    import org.apache.spark.SparkConf
    
    object MyFirst {
      def main(args: Array[String]) {
        // create Spark context with Spark configuration
        val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))
        // more code ...
      }
    }
    

    And then use:

    spark-submit MyFirst --class com.testing.MyFirst /home/ram/Downloads/sbt/src/target/scala-2.10/MyFirst_2.10-0.1.0-SNAPSHOT.jar
    

    Also make sure you have created the MANIFEST.MF file.

    Here's a brief example of what would it look like:

    Manifest-Version: 1.0
    Main-Class: com.testing.MyFirst