Fehler beim Importieren von Pandas in Spark-Submit Script

Question

Ich benutze Spark-Submit, und ich bekomme Fehler, wenn ich das Paket 'Pandas' importiere.

Von Notebooks kann ich Pandas ohne Probleme importieren.

Aber wenn ich die folgende Datei durch einreichen funken einreichen:

from pymongo import MongoClient 
import pandas as pd 

mongo_url = "mongodb://..." 

LOGDB = MongoClient(mongo_url).schedule.log_test 
LOGDB.insert({'m': 'Worked Spark Submit Test'}) 

ich die folgende Fehlermeldung erhalten:

stderr_1461683917691631694 Datei:

Spark Command: /usr/local/src/spark160master/ibm-java-x86_64-80//bin/java -cp /usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-launcher_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-network-shuffle_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/gson-2.2.4.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/guava-14.0.1.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/Java-WebSocket-1.3.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-ego_2.10-1.6.0.jar:/usr/local/src/spark160master/spark/profile/batch/:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/spark-assembly-1.6.0-hadoop2.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-api-jdo-3.2.6.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-core-3.2.10.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-rdbms-3.2.9.jar:/usr/local/src/data-connectors-1.4.1/* -Dspark.eventLog.enabled=true -Dspark.master=spark://yp-spark-dal09-env5-0005:7083 -Dspark.executor.memory=1024m -Dspark.executor.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.app.name=test_submit.py -Dspark.executor.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.files=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py -Dspark.eventLog.dir=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/events -Dspark.files.useFetchCache=false -Dspark.shuffle.service.port=7340 -Xms512m -Xmx512m org.apache.spark.deploy.ego.EGOClusterDriverWrapper {{WORKER_URL}} /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py org.apache.spark.deploy.PythonRunner --primary-py-file test_submit.py 
======================================== 
log4j:ERROR Could not find value for key log4j.appender.FILE 
log4j:ERROR Could not instantiate appender named "FILE". 
16/04/26 10:18:43 INFO deploy.ego.EGOClusterDriverWrapper: Registered signal handlers for [TERM, HUP, INT] 
16/04/26 10:18:44 WARN hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:44 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a) 
16/04/26 10:18:45 INFO spark.util.Utils: Successfully started service 'EGOClusterDriverWrapper-driver-20160426101839-0010-41cca182-f2d6-408c-81c0-ec288530f82e' on port 55398. 
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:45 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a) 
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Fetching file from /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py 
16/04/26 10:18:45 INFO spark.util.Utils: Copying /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py 
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Starting the user JAR in a separate Thread 
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Waiting for spark context initialization ... 0 
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Final app status: 1, exitCode: 15, (reason: User class threw exception: User application exited with 1) 
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Sending driver program state to master 
Exception in thread "Driver" org.apache.spark.SparkUserAppException: User application exited with 1 
     at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:88) 
     at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) 
     at java.lang.reflect.Method.invoke(Method.java:507) 
     at org.apache.spark.deploy.ego.EGOClusterDriverWrapper$$anon$3.run(EGOClusterDriverWrapper.scala:434) 

stdout_1461683917691631694

no extra config 
load default config from : /usr/local/src/spark160master/spark/profile/batch/ 
/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String 
Traceback (most recent call last): 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py", line 19, in <module> 
    import pandas as pd 
    File "/usr/lib64/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/__init__.py", line 6, in <module> 
    from . import hashtable, tslib, lib 
    File "numpy.pxd", line 157, in init pandas.hashtable (pandas/hashtable.c:22150) 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/__init__.py", line 180, in <module> 
    from . import add_newdocs 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module> 
    from numpy.lib import add_newdoc 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module> 
    from .type_check import * 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module> 
    import numpy.core.numeric as _nx 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/__init__.py", line 14, in <module> 
    from . import multiarray 
ImportError: /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String 

Was mache ich falsch?

Answer 1

Ich kann Ihren Fehler nicht genau reproduzieren, weil ich nicht Ihren gesamten Jobcode für das Senden von Code-Codes habe.

Initialisieren Sie Ihren Funkenkontext?

Hier ist mein Code, das funktioniert: -

from pymongo import MongoClient

import pandas as pd

from pyspark import SparkContext

if __name__ == "__main__":

sc = SparkContext(appName="MongoDB CON TEST") 
    print("Hello Spark Demo for Mongodb connection") 
    print("Begining MongoCon") 

    mongo_url = "mongodb://ds045252.mlab.com:45252" 

    connection = MongoClient(mongo_url) 

    db = connection["samplemongodb"] 
    # MongoLab has user authentication 
    db.authenticate("username", "password") 
    db.Photos.insert({'m': 'Worked Spark Submit Test Real test'}) 
    print("Insert worked") 
    sc.stop() 

vollständigen Code finden Sie hier in Github mit Vertiefung: - https://github.com/charles2588/sparkjobs/blob/master/mongodbtest.py

Senden Sie den Auftrag mit diesem: -

./spark-submit.sh --vcap credentials/vcap.json --deploy-mode cluster --master https://169.54.219.20 mongodbtest.py

Danke,

Charles.

Answer 2

Notebooks haben Pandas und NumPy vorinstalliert. Ihr Stacktrace zeigt an, dass spark-submit versucht, NumPy von .local zu laden, also handelt es sich um eine Version, die Sie selbst installiert haben. Ist nicht ein vorinstalliertes NumPy in der Spark-Submission-Umgebung vorhanden, das Sie verwenden könnten?

Der Fehler wird wahrscheinlich durch eine falsche Einstellung von LD_LIBRARY_PATH verursacht. Haben Sie das lokale NumPy von einem Notebook aus installiert und versuchen nun, es aus der Funkebene zu verwenden? Dies sind unterschiedliche Ausführungsumgebungen, die möglicherweise nicht die gleiche Einstellung für LD_LIBRARY_PATH haben.