3
Hey alles, ich habe eine einfache 2 Knoten Cluster 1 Master und 1 Slave (Hadoop und Funken). Alles funktionierte gut, aber aus irgendeinem Grund kann ich jetzt, wenn ich den Master-Funke starte, keine Verbindung zum Master-Web-Interface herstellen (Port 8081 setze ich explizit in der Datei spark-env.sh).Apache Funken Master UI funktioniert nicht
Spark Command: /usr/lib/jvm/java-8-oracle/bin/java -cp /usr/local/hive/lib/*:/usr/local/spark/conf/:/usr/local/spark/assembly/target/scala-2.11/jars/*:/usr/local/hadoop/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host ip-172-31-18-164 --port 7077 --webui-port 8081
========================================
16/07/23 00:47:55 INFO master.Master: Started daemon with process name: [email protected]
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for TERM
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for HUP
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for INT
16/07/23 00:47:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls groups to:
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls groups to:
16/07/23 00:47:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); groups with view permissions: Set(); users with modify permissions: Set(ubuntu); groups with modify permissions: Set()
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
16/07/23 00:47:56 INFO master.Master: Starting Spark master at spark://ip-172-31-18-164:7077
16/07/23 00:47:56 INFO master.Master: Running Spark version 2.1.0-SNAPSHOT
16/07/23 00:47:56 INFO util.log: Logging initialized @1447ms
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/app,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/app/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/static,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/app/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/driver/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8081}
16/07/23 00:47:56 INFO server.Server: Started @1558ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'MasterUI' on port 8081.
16/07/23 00:47:56 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://172.31.18.164:8081
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{ip-172-31-18-164:6066}
16/07/23 00:47:56 INFO server.Server: Started @1587ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service on port 6066.
16/07/23 00:47:56 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/metrics/master/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/metrics/applications/json,null,AVAILABLE}
16/07/23 00:47:56 INFO master.Master: I have been elected leader! New state: ALIVE
16/07/23 00:47:57 WARN server.HttpChannel:/
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)
at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.spark_project.jetty.server.Server.handle(Server.java:499)
at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool:
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getStatus()I
at org.spark_project.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:112)
at org.spark_project.jetty.server.Response.sendError(Response.java:597)
at org.spark_project.jetty.server.HttpChannel.handleException(HttpChannel.java:487)
at org.spark_project.jetty.server.HttpConnection$HttpChannelOverHttp.handleException(HttpConnection.java:594)
at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:387)
at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool: Unexpected thread death: [email protected] in MasterUI{STARTED,8<=8<=200,i=5,q=0}
16/07/23 00:47:58 INFO master.Master: Registering worker 172.31.7.238:60863 with 2 cores, 6.0 GB RAM
ja, ich erkennen, dass ich eine * in meinem SPARK_CLASSPATH im Hive Lib tut die mysql.jar zu bekommen, aber dann in einer anderen Version des Servlet es brachte in Ich habe es jetzt explizit die mysql.jar-Datei aufrufen. – Neil