java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError Ausnahmen bei der Ausführung von Kinesis Funke Job

Question

Ich habe schwer versucht, herauszufinden, meine Scala-Abhängigkeiten spark (2.1.0) Job.

Meine build.sbt Datei:

name := "test" 

version := "0.0.1" 

scalaVersion := "2.11.0" 

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0" % "provided" 
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided" 
libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % "2.1.0" 
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.5.1" 

assemblyJarName in assembly := "test.jar" 


mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => 
    { 
    case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard 
    case m if m.startsWith("META-INF") => MergeStrategy.discard 
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first 
    case PathList("org", "apache", xs @ _*) => MergeStrategy.first 
    case PathList("org", "jboss", xs @ _*) => MergeStrategy.first 
    case "about.html" => MergeStrategy.rename 
    case "reference.conf" => MergeStrategy.concat 
    case _ => MergeStrategy.first 
    } 
} 

exportJars:= true 

mainClass in assembly := Some("test.Job") 
``` 

Wenn ich meinen Job ausführen es java.lang.NoSuchMethodError Ausnahmen auslöst.

17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 87 
17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 86 
17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 85 
17/03/08 05:19:15 ERROR worker.Worker: Worker.run caught exception, sleeping for 1000 milli seconds! 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long; 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:156) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.consumeShard(ShardConsumer.java:125) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:335) 
    at org.apache.spark.streaming.kinesis.KinesisReceiver$$anon$1.run(KinesisReceiver.scala:174) 
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long; 
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) 
    at java.util.concurrent.FutureTask.get(FutureTask.java:192) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:136) 
    ... 3 more 
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long; 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResultAndRecordMillisBehindLatest(ProcessTask.java:291) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResult(ProcessTask.java:249) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:120) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

Answer 1

laufen Sie auf aws emr? Wenn ja die folgenden Links helfen können:

Spark streaming 1.6.1 is not working with Kinesis asl 1.6.1 and asl 2.0.0-preview http://www.waitingforcode.com/apache-spark/shading-solution-dependency-hell-spark/read

im Wesentlichen aws EMR unterstützt protobuf 2.5, aber der Funke-Kern, funken Streaming und funken Streaming-Kinesis-asl Versionen alle hängen von protobuf 2.6.1, die Art und Weise, wie wir dieses Problem gelöst haben, als wir es gefunden haben, war durch ein schattiertes Glas, und die zwei Links oben geben gute Beispiele, wie man das aufstellt.