0
Ich denke, es hat etwas mit Speicher zu tun, weil es für kleinere Datensätze funktionierte. Das Programm verwendet und wird vorzeitig beendet, während die logistische Regression von Spark-Mllib verwendet wird. Ich führe diesen Befehl aus, um mein Spark-Programm auf HDFS zu starten.Warum wird SparkContext während der logistischen Regression heruntergefahren?
export SPARK_CONF_DIR=/home/gs/conf/spark/latest
export SPARK_HOME=/home/gs/spark/latest
$SPARK_HOME/bin/spark-submit --class algoRunner --master yarn --deploy-mode cluster --conf spark.dynamicAllocation.enabled=true \
--executor-memory 8g --queue default --conf spark.hadoop.hive.querylog.location='${java.io.tmpdir}/hivelogs' \
~/spark/Product-Classifier-Pipeline-assembly-1.0.jar
Ich erhalte den folgenden Fehler:
17/08/02 21:53:40 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
17/08/02 21:53:40 INFO SparkContext: Invoking stop() from shutdown hook
17/08/02 21:53:40 INFO SparkUI: Stopped Spark web UI at http://gsrd219n01.red.ygrid.yahoo.com:45546
17/08/02 21:53:40 INFO DAGScheduler: Job 10 failed: treeAggregate at LogisticRegression.scala:1670, took 2.351935 s
17/08/02 21:53:40 INFO DAGScheduler: ShuffleMapStage 19 (treeAggregate at LogisticRegression.scala:1670) failed in 1.947 s due to Stage cancelled because SparkContext was shut down
17/08/02 21:53:40 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted([email protected])
17/08/02 21:53:40 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(10,1501710820713,JobFailed(org.apache.spark.SparkException: Job 10 cancelled because SparkContext was shut down))
17/08/02 21:53:40 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job 10 cancelled because SparkContext was shut down
org.apache.spark.SparkException: Job 10 cancelled because SparkContext was shut down