Kommunikation mit Spark mit Spark JobServer im Docker

Ich habe Funf-Cluster im Single-Modus gestartet.Kommunikation mit Spark mit Spark JobServer im Docker

Ich kann Job mit Spark-Core-lib für Scala erfolgreich ausführen. Ich möchte Spark JobServer für die Verwaltung von Jobs verwenden. Ich begann es in Docker auf dem Master-Host-Knoten:

docker run -d -p 8090:8090 -e SPARK_MASTER=spark://10.0.0.56:7077 velvia/spark-jobserver:0.6.0

Dann versuche ich bereits hinzugefügt Job auszuführen (I durch UI verifiziert, dass es hinzugefügt wurde):

curl-d "input.string =somedata" '10.0.0.56:8090/jobs?appName=myjob&classPath=sparkscala.Job'

Antwort:

{ 
    "status": "ERROR", 
    "result": { 
    "message": "Futures timed out after [15 seconds]", 
    "errorClass": "java.util.concurrent.TimeoutException", 
    "stack": ["scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)", "scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)", "scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)", "akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoo 
lBuilder.scala:169)", "scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)", "akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)", "akka.dispatch.BatchingExecutor$Batch.blockOn(BatchingExecutor.scala:101)", "scala.concurrent.Await$.result(packa 
ge.scala:107)", "spark.jobserver.WebApi.spark$jobserver$WebApi$$getJobManagerForContext(WebApi.scala:478)", "spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$33$$anonfun$apply$34.apply(WebApi.scala:419)", "spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$33$$anonfun$apply$34.apply(WebApi.scala:412 
)", "spray.routing.ApplyConverterInstances$$anon$18$$anonfun$apply$5.apply(ApplyConverterInstances.scala:53)", "spray.routing.ApplyConverterInstances$$anon$18$$anonfun$apply$5.apply(ApplyConverterInstances.scala:52)", "spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive. 
scala:38)", "spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.scala:37)", "spray.routing.directives.BasicDirectives$$anon$2.happly(BasicDirectives.scala:79)", "spray.routing.Directive$$anon$7$$anonfun$happly$4.apply(Directive.scala:86)", "spray.routing.Directive$$anon 
$7$$anonfun$happly$4.apply(Directive.scala:86)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly 
$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(Bas 
icDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives 
.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$a 
pply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$a 
nonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$an 
on$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.rout 
ing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$m 
cVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(U 
nregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3 
$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$Rout 
eConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialF 
unction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.Requ 
estContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDire 
ctives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$ti 
lde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartial 
Function$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", " 
akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.direct 
ives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.a 
pply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anon 
fun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPar 
tialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scal 
a:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteCo 
ncatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.R 
outeConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.rou 
ting.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirective 
s.scala:81)", "spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirectives.scala:79)", "scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)", "akka.dispatch.Ba 
tchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)", "scala.concurrent.BlockContext$.withBlockContext(BlockContext.sc 
ala:72)", "akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)", "akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)", "akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)", "scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)", "s 
cala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)", "scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)", "scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)"] 
    }

Logs auf Job Server:

[2016-05-05 13:27:00,655] INFO ient.AppClient$ClientActor [] [] - Connecting to master akka.tcp://[email protected]:7077/user/Master... 
[2016-05-05 13:27:14,457] ERROR ocalContextSupervisorActor [] [] - Exception after sending Initialize to JobManagerActor 
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://JobServer/user/context-supervisor/c7f81d9f-sparkscala.Server#-306023562]] after [15000 ms] 
    at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) 
    at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) 
    at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694) 
    at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691) 
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) 
    at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) 
    at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) 
    at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) 
    at java.lang.Thread.run(Thread.java:745)

Spark-Version: 1.6.1 Vorgefertigte für Hadoop 2.6
Spark-Jobserver: 0.6.0
Job Scala Version 2.10.6

Quelle

2016-05-05 Cortwave

ich las Version von Job Server aktualisiert (0.6.2.mesos-0.28.1.spark -1.6.1) und alles ist jetzt gut. Möglicherweise war eine Kollision mit Spark-Versionen.

Quelle

2016-05-06 12:13:21 Cortwave

Funktioniert das, wenn sjs außerhalb des Master-Host-Knotens gehostet wird? Ich habe diese Szenarien und Spark-Worker verbindet zurück zu Docker-Container-IP, die nicht erreichbar ist. – noorul

Es sollte remote funktionieren, sollten Sie nur Ports binden und korrekte externe ips in allen Konfigurationen angeben – Cortwave

wenn ich docker Host IP in spark.driver.host angeben, Job kann nicht gestartet werden, da es versucht, an einen zufälligen Port auf Docker Host-IP zu binden . – noorul

Kommunikation mit Spark mit Spark JobServer im Docker

Antwort

Verwandte Themen