Fehler beim Erstellen von Wordcount-Projekt mit Spark & ​​Java auf Eclipse in Cloudera durch VMware

Question

Ich versuche Wordcount Prject mit Spark & Java auf Eclipse in Cloudera über VMware zu erstellen. Die Java-Version ist 1.7 und Spark-Version ist 2.0.0. Der Code innerhalb „JavaWordCount.java“ Klasse im Projekt ist wie folgt:

package com.vishal.wc; 

    import scala.Tuple2; 


import org.apache.hadoop.hive.ql.exec.spark.session.SparkSession; 
import org.apache.spark.api.java.JavaRDD; 

    public class JavaWordCount { 

     public static final Pattern SPACE = Pattern.compile(" "); 

public static void main(String[] args) throws Exception { 

       if(args.length < 2){   
System.err.println("Usage: JavaWordCount <InputFile> <OutputFile>");  System.exit(1);  }  
SparkSession spark= SparkSession.builder().appName("JavaWordCount").getOrCreate();  JavaRDD<String> lines = spark.read().textFile(args[0]).javaRDD();  JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>(){ 

public Iterator<String> call(String s){    
return Arrays.asList(s.split(" ")).iterator();  
} 
});   
JavaPairRDD<String, Integer> ones = words.mapToPair(new PairFunction<String, String, Integer>(){ 

public tuple2<String, Integer> call(String s){ 

return new tuple2<>(s,1);   
} 
}); 

JavaPairRDD<String, Integer> counts = ones.reduceByKey(
new Function2<Integer, Integer, Integer>(){ 

public Integer call(Integer i1, Integer i2){ 
         return i1 = i2; 
        }   
}); 
counts.saveAsTextFile(args[1]);   
spark.stop();  
} 
} 

Fehler aufgetreten sind, da es keine Funken Gläser gegeben. Ich habe Gläser aus Spark-2.0.0-bin-hadoop-2.7.tgz in den Build-Pfad eingefügt, aber die Fehler sind fast gleich. Fehler sind unten angegeben:

Description Resource Path Location Type 
FlatMapFunction cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 26 Java Problem 
Function2 cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 44 Java Problem 
Iterator cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 28 Java Problem 
JavaPairRDD cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 32 Java Problem 
JavaPairRDD cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 42 Java Problem 
PairFunction cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 32 Java Problem 
The method builder() is undefined for the type SparkSession JavaWordCount.java /SparkProject/src/com/vishal/wc line 22 Java Problem 
The method flatMap(FlatMapFunction<String,U>) in the type AbstractJavaRDDLike<String,JavaRDD<String>> is not applicable for the arguments (new FlatMapFunction<String,String>(){}) JavaWordCount.java /SparkProject/src/com/vishal/wc line 26 Java Problem 
The method mapToPair(PairFunction<String,K2,V2>) in the type AbstractJavaRDDLike<String,JavaRDD<String>> is not applicable for the arguments (new PairFunction<String,String,Integer>(){}) JavaWordCount.java /SparkProject/src/com/vishal/wc line 32 Java Problem 
The method read() is undefined for the type SparkSession JavaWordCount.java /SparkProject/src/com/vishal/wc line 24 Java Problem 
The method stop() is undefined for the type SparkSession JavaWordCount.java /SparkProject/src/com/vishal/wc line 52 Java Problem 
tuple2 cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 35 Java Problem 
tuple2 cannot be resolved to a type JavaWordCount.java /SparkProject/src/com/vishal/wc line 37 Java Problem 

Bitte helfen.

Answer 1

Sie müssen wie die fehlenden Bibliotheken importieren unter

import org.apache.spark.api.java.JavaPairRDD; 
import org.apache.spark.api.java.function.FlatMapFunction; 
import org.apache.spark.api.java.function.Function2; 
import org.apache.spark.api.java.function.PairFunction; 

Eclipse-Verknüpfung Ctrl + Shift + O bietet für alle fehlenden Importe zu bekommen.