Verwende ich mehrere Scala-Versionen?

Question

Ich bin neu in Scala und Spark. Ich machte den einfachen Code und es lief erfolgreich auf meinem lokalen Rechner. Also, ich machte die .jar Dateien mit Maven und kopierte sie in meine Cluster-Maschine, um es auf dem verteilten System zu testen. Ich habe jedoch meinen Befehl gestartet, die Konsole löst den Fehler wie folgt aus.

*******CLASSPATH = ******** 
java.lang.ClassNotFoundException: App 
     at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
     at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
     at java.lang.Class.forName0(Native Method) 
     at java.lang.Class.forName(Class.java:274) 
     at org.apache.spark.util.Utils$.classForName(Utils.scala:225) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:693) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

ich gegoogelt es bereits festgestellt, dass der Klassenname package name + class name sein sollte. Aber für meinen Fall hat es nicht funktioniert. Und ich fand eine andere Ursache, dass ich mehrere Scala-Versionen verwenden könnte. Also habe ich meine pom.xml überprüft, um meine scala und Funken Versionen zu gewährleisten. Ich änderte den Versionsnamen entsprechend dem Gebrauch des Versionsclusters, aber das Ergebnis war das gleiche.

Unten ist meine pom.xml.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>com.sclee.scala0</groupId> 
    <artifactId>scala_tutorial</artifactId> 
    <version>1.0-SNAPSHOT</version> 
    <name>${project.artifactId}</name> 
    <description>My wonderfull scala app</description> 
    <inceptionYear>2015</inceptionYear> 
    <licenses> 
    <license> 
     <name>My License</name> 
     <url>http://....</url> 
     <distribution>repo</distribution> 
    </license> 
    </licenses> 

    <properties> 
    <maven.compiler.source>1.6</maven.compiler.source> 
    <maven.compiler.target>1.6</maven.compiler.target> 
    <encoding>UTF-8</encoding> 
    <scala.version>2.10.4</scala.version> 
    <scala.compat.version>2.10</scala.compat.version> 
    </properties> 

    <dependencies> 


    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10 --> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-mllib_2.10</artifactId> 
     <version>2.0.1</version> 
    </dependency> 


    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.10</artifactId> 
     <version>2.0.1</version> 
    </dependency> 


    <!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library --> 
    <dependency> 
     <groupId>org.scala-lang</groupId> 
     <artifactId>scala-library</artifactId> 
     <version>2.10.4</version> 
    </dependency> 


    </dependencies> 


    <pluginRepositories> 
    <pluginRepository> 
     <id>scala-tools.org</id> 
     <name>Scala-tools Maven2 Repository</name> 
     <url>http://scala-tools.org/repo-releases</url> 
    </pluginRepository> 
    </pluginRepositories> 


    <build> 
    <plugins> 
     <plugin> 
     <groupId>org.scala-tools</groupId> 
     <artifactId>maven-scala-plugin</artifactId> 
     <executions> 

      <execution> 
      <id>compile</id> 
      <goals> 
       <goal>compile</goal> 
      </goals> 
      <phase>compile</phase> 
      </execution> 
      <execution> 
      <id>test-compile</id> 
      <goals> 
       <goal>testCompile</goal> 
      </goals> 
      <phase>test-compile</phase> 
      </execution> 
      <execution> 
      <phase>process-resources</phase> 
      <goals> 
       <goal>compile</goal> 
      </goals> 
      </execution> 
     </executions> 
     </plugin> 
     <plugin> 
     <artifactId>maven-compiler-plugin</artifactId> 
     <configuration> 
      <source>1.5</source> 
      <target>1.5</target> 
     </configuration> 
     </plugin> 
    </plugins> 
    </build> 

</project> 

Und wenn ich zu reinigen versucht, kompilieren und Paket mit Maven Paket, gibt es einige gefunden Warnungen. (Ich bin nicht sicher, ob diese Warnungen für mein Ergebnis relevant sein könnten. Um dieses Problem zu beheben, habe ich die Protokollnachricht wie folgt angehängt).

/usr/local/java/jdk1.7.0_80/bin/java -Dmaven.home=/usr/local/maven/apache-maven-3.1.1 -Dclassworlds.conf=/usr/local/maven/apache-maven-3.1.1/bin/m2.conf -Didea.launcher.port=7536 -Didea.launcher.bin.path=/usr/local/intellij/idea-IC-163.10154.41/bin -Dfile.encoding=UTF-8 -classpath /usr/local/maven/apache-maven-3.1.1/boot/plexus-classworlds-2.5.1.jar:/usr/local/intellij/idea-IC-163.10154.41/lib/idea_rt.jar com.intellij.rt.execution.application.AppMain org.codehaus.classworlds.Launcher -Didea.version=2016.3.2 clean compile package 
[INFO] Scanning for projects... 
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.sclee.scala0:scala_tutorial:jar:1.0-SNAPSHOT 
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 92, column 15 
[WARNING] 'build.plugins.plugin.version' for org.scala-tools:maven-scala-plugin is missing. @ line 65, column 15 
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. 
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects. 
[WARNING] 
[INFO]                   
[INFO] ------------------------------------------------------------------------ 
[INFO] Building scala_tutorial 1.0-SNAPSHOT 
[INFO] ------------------------------------------------------------------------ 
Downloading: http://scala-tools.org/repo-releases/org/scala-tools/maven-scala-plugin/maven-metadata.xml 
[WARNING] Could not transfer metadata org.scala-tools:maven-scala-plugin/maven-metadata.xml from/to scala-tools.org (http://scala-tools.org/repo-releases): peer not authenticated 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ scala_tutorial --- 
[INFO] Deleting /home/dst/Documents/01_Intellij_workspace/scala_tutorial/target 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ scala_tutorial --- 
[INFO] Using 'UTF-8' encoding to copy filtered resources. 
[INFO] skip non existing resourceDirectory /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/main/resources 
[INFO] 
[INFO] --- maven-scala-plugin:2.15.2:compile (default) @ scala_tutorial --- 
[INFO] Checking for multiple versions of scala 
[WARNING] Expected all dependencies to require Scala version: 2.10.4 
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5 
[WARNING] Multiple versions of scala libraries detected! 
[INFO] includes = [**/*.scala,**/*.java,] 
[INFO] excludes = [] 
[INFO] /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/main/scala:-1: info: compiling 
[INFO] Compiling 1 source files to /home/dst/Documents/01_Intellij_workspace/scala_tutorial/target/classes at 1486089232696 
[WARNING] warning: there were 2 deprecation warning(s); re-run with -deprecation for details 
[WARNING] one warning found 
[INFO] prepare-compile in 0 s 
[INFO] compile in 6 s 
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ scala_tutorial --- 
[INFO] Nothing to compile - all classes are up to date 
[INFO] 
[INFO] --- maven-scala-plugin:2.15.2:compile (compile) @ scala_tutorial --- 
[INFO] Checking for multiple versions of scala 
[WARNING] Expected all dependencies to require Scala version: 2.10.4 
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5 
[WARNING] Multiple versions of scala libraries detected! 
[INFO] includes = [**/*.scala,**/*.java,] 
[INFO] excludes = [] 
[INFO] Nothing to compile - all classes are up to date 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ scala_tutorial --- 
[INFO] Using 'UTF-8' encoding to copy filtered resources. 
[INFO] skip non existing resourceDirectory /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/main/resources 
[INFO] 
[INFO] --- maven-scala-plugin:2.15.2:compile (default) @ scala_tutorial --- 
[INFO] Checking for multiple versions of scala 
[WARNING] Expected all dependencies to require Scala version: 2.10.4 
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5 
[WARNING] Multiple versions of scala libraries detected! 
[INFO] includes = [**/*.scala,**/*.java,] 
[INFO] excludes = [] 
[INFO] Nothing to compile - all classes are up to date 
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ scala_tutorial --- 
[INFO] Nothing to compile - all classes are up to date 
[INFO] 
[INFO] --- maven-scala-plugin:2.15.2:compile (compile) @ scala_tutorial --- 
[INFO] Checking for multiple versions of scala 
[WARNING] Expected all dependencies to require Scala version: 2.10.4 
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5 
[WARNING] Multiple versions of scala libraries detected! 
[INFO] includes = [**/*.scala,**/*.java,] 
[INFO] excludes = [] 
[INFO] Nothing to compile - all classes are up to date 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ scala_tutorial --- 
[INFO] Using 'UTF-8' encoding to copy filtered resources. 
[INFO] skip non existing resourceDirectory /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/test/resources 
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ scala_tutorial --- 
[INFO] No sources to compile 
[INFO] 
[INFO] --- maven-scala-plugin:2.15.2:testCompile (test-compile) @ scala_tutorial --- 
[INFO] Checking for multiple versions of scala 
[WARNING] Expected all dependencies to require Scala version: 2.10.4 
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5 
[WARNING] Multiple versions of scala libraries detected! 
[INFO] includes = [**/*.scala,**/*.java,] 
[INFO] excludes = [] 
[WARNING] No source files found. 
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ scala_tutorial --- 
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ scala_tutorial --- 
[INFO] Building jar: /home/dst/Documents/01_Intellij_workspace/scala_tutorial/target/scala_tutorial-1.0-SNAPSHOT.jar 
[INFO] ------------------------------------------------------------------------ 
[INFO] BUILD SUCCESS 
[INFO] ------------------------------------------------------------------------ 
[INFO] Total time: 15.990s 
[INFO] Finished at: Thu Feb 02 21:34:00 EST 2017 
[INFO] Final Memory: 27M/291M 
[INFO] ------------------------------------------------------------------------ 

Process finished with exit code 0 

Der Scala-Version-Cluster verwendet ist 2.10.4. Also habe ich versucht, alle Versionen in der pom.xml zu ändern. Und der Scala-Code ist wie folgt. Es ist eine einfache Aufgabe, meine Daten mit DataFrame zu transformieren.

Was ich meine von Scala geschriebenen JAR-Dateien verwenden und im Cluster-Modus testen möchte. Und meine Fragen sind, dass es irgendeinen falschen Prozess (Versionsausgabe oder irgendetwas) gibt, dass du komisch aussiehst? Jede Hilfe wird geschätzt.

package com.sclee.scala0 
import org.apache.spark._ 
import org.apache.spark.sql.SQLContext 


object App { 
    def main(args: Array[String]): Unit = { 
    val conf = new SparkConf().setAppName("wordCount").setMaster("local[*]") 
    // Create a Scala Spark Context. 
    val sc = new SparkContext(conf) 

    val sqlContext= new org.apache.spark.sql.SQLContext(sc) 
    import sqlContext.implicits._ 


    val df = sc.textFile(args(0)).map(_.split('\t')).map(x => (x(0),x(1),x(2),x(3),x(4),x(5),x(6))).toDF("c1","c2","c3","c4","c5","c6","c7") 
    val res = df.explode("c7","c8")((x:String) => x.split('|')).drop("c7") 

    res.write.format("com.databricks.spark.csv").option("delimiter","\t").save(args(1)) 
    } 
} 

Und schließlich ist dies mein Befehl, scala jar auszuführen.

spark-submit \ 
     --class App \ 
     --master spark://spark.dso.xxxx \ 
     --executor-memory 10G \ 
     /home/jumbo/user/sclee/dt/jars/scala_tutorial-1.0-SNAPSHOT.jar \ 
     /user/sclee/data_big/ /user/sclee/output_scala_csv 

Answer 1

Scala Objektnamen sollten mit Paketnamen zugeordnet werden, während die scala-Funkenjob übergeben, und deshalb wird Ihre --class Konfiguration:

--class com.sclee.scala0.App 

Verwenden oben Konfiguration während der Einreichung des Antrags und es wird beseitigt dein Fehler.

Aktualisieren Sie Ihre Maven pom.xml Abhängigkeitsabschnitt mit folgenden:

<dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-mllib_2.10</artifactId> 
     <version>${scala.version}</version> 
    </dependency> 


    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.10</artifactId> 
     <version>${scala.version}</version> 
    </dependency> 

Es ermöglicht Ihre pom.xml Ihre Scala Version kompatibel Gläser von Remote-Repository zum Download bereit.

Ich hoffe, es wird helfen.