Mrjob im Hadoop-Modus: Fehler beim Starten des Jobs, ungültiger Eingabepfad: Datei nicht vorhanden

Question

Ich versuche, die Mrjob example aus dem Buch Hadoop mit Python auf meinem Laptop, im pseudo-verteilten Modus auszuführen.

So (die Datei salaries.csv kann here zu finden) ich die NameNode und die DataNode starten:

start-dfs.sh 

kehrt:

Starting namenodes on [localhost] 
localhost: starting namenode, logging to /home/me/hadoop-2.7.3/logs/hadoop-me-namenode-me-Notebook-PC.out 
localhost: starting datanode, logging to /home/me/hadoop-2.7.3/logs/hadoop-me-datanode-me-Notebook-PC.out 
Starting secondary namenodes [0.0.0.0] 
0.0.0.0: starting secondarynamenode, logging to /home/me/hadoop-2.7.3/logs/hadoop-me-secondarynamenode-me-Notebook-PC.out 

Ich habe auch kein Problem Erstellen der Eingabedateistruktur und Kopieren salaries.csv in die hdfs:

hdfs dfs -mkdir /user/ 
hdfs dfs -mkdir /user/me/ 
hdfs dfs -mkdir /user/me/input/ 
hdfs dfs -put /home/me/Desktop/work/cv/hadoop/salaries.csv /user/me/input/ 
hdfs dfs -ls /user/me/input/ 

kehrt:

Found 1 items 
-rw-r--r-- 3 me supergroup 1771685 2016-12-24 15:57 /user/me/input/salaries.csv 

ich auch top_salaries.py ausführbar machen:

sudo chmod a+x /home/me/Desktop/work/cv/hadoop/top_salaries.py 

top_salaries.py im lokalen Modus lauching funktioniert auch:

python2 top_salaries.py -r local salaries.csv > answer.csv 

kehrt:

No configs found; falling back on auto-configuration 
Creating temp directory /tmp/top_salaries.me.20161224.195052.762894 
Running step 1 of 1... 
Counters: 1 
    warn 
     missing gross=3223 
Counters: 1 
    warn 
     missing gross=3223 
Streaming final output from /tmp/top_salaries.me.20161224.195052.762894/output... 
Removing temp directory /tmp/top_salaries.me.20161224.195052.762894... 

jedoch läuft diese Arbeit auf dem hadoop python2 top_salaries.py -r hadoop hdfs:///user/me/input/salaries.csv kehrt (Dinge zusammenfügen):

No configs found; falling back on auto-configuration 
Looking for hadoop binary in $PATH... 
Found hadoop binary: /home/me/hadoop-2.7.3/bin/hadoop 
Using Hadoop version 2.7.3 
Looking for Hadoop streaming jar in /home/me/hadoop-2.7.3... 
Found Hadoop streaming jar: /home/me/hadoop-2.7.3/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar 
Creating temp directory /tmp/top_salaries.me.20161224.195201.967990 
Copying local files to hdfs:///user/me/tmp/mrjob/top_salaries.me.20161224.195201.967990/files/... 
Running step 1 of 1... 
    session.id is deprecated. Instead, use dfs.metrics.session-id 
    Initializing JVM Metrics with processName=JobTracker, sessionId= 
    Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
    Cleaning up the staging area file:/tmp/hadoop-me/mapred/staging/me553683497/.staging/job_local553683497_0001 
    Error launching job , bad input path : File does not exist: /tmp/hadoop-me/mapred/staging/me553683497/.staging/job_local553683497_0001/files/mrjob.zip#mrjob.zip 
    Streaming Command Failed! 
Attempting to fetch counters from logs... 
Can't fetch history log; missing job ID 
No counters found 
Scanning logs for probable cause of failure... 
Can't fetch history log; missing job ID 
Can't fetch task logs; missing application ID 
Step 1 of 1 failed: Command '['/home/me/hadoop-2.7.3/bin/hadoop', 'jar', '/home/me/hadoop-2.7.3/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar', '-files', 'hdfs:///user/me/tmp/mrjob/top_salaries.me.20161224.195201.967990/files/mrjob.zip#mrjob.zip,hdfs:///user/me/tmp/mrjob/top_salaries.me.20161224.195201.967990/files/setup-wrapper.sh#setup-wrapper.sh,hdfs:///user/me/tmp/mrjob/top_salaries.me.20161224.195201.967990/files/top_salaries.py#top_salaries.py', '-input', 'hdfs:///user/me/input/salaries.csv', '-output', 'hdfs:///user/me/tmp/mrjob/top_salaries.me.20161224.195201.967990/output', '-mapper', 'sh -ex setup-wrapper.sh python top_salaries.py --step-num=0 --mapper', '-combiner', 'sh -ex setup-wrapper.sh python top_salaries.py --step-num=0 --combiner', '-reducer', 'sh -ex setup-wrapper.sh python top_salaries.py --step-num=0 --reducer']' returned non-zero exit status 512 

Edit:

dies mein Kern-site.xml ist:

<configuration> 
<property>   
    <name>fs.defaultFS</name>   
    <value>hdfs://localhost:9000</value>  
</property> 
</configuration> 

und das ist meine hdfs-site.xml:

<configuration> 
    <property> 
     <name>dfs.namenode.name.dir</name> 
     <value>/home/me/Desktop/work/cv/hadoop/namenode</value> 
    </property> 
    <property> 
     <name>dfs.datanode.data.dir</name> 
     <value>/home/me/Desktop/work/cv/hadoop/datanode</value> 
    </property> 
</configuration> 

(die anderen XML-Konfigurationsdateien, ich habe bearbeitet/geändert nicht)

Edit2:

hier ist den Python-Skript (die gleichen wie auf dem Github Link oben)

from mrjob.job import MRJob 
from mrjob.step import MRStep 
import csv 

cols = 'Name,JobTitle,AgencyID,Agency,HireDate,AnnualSalary,GrossPay'.split(',') 

class salarymax(MRJob): 

    def mapper(self, _, line): 
     # Convert each line into a dictionary 
     row = dict(zip(cols, [ a.strip() for a in csv.reader([line]).next()])) 

     # Yield the salary 
     yield 'salary', (float(row['AnnualSalary'][1:]), line) 

     # Yield the gross pay 
     try: 
      yield 'gross', (float(row['GrossPay'][1:]), line) 
     except ValueError: 
      self.increment_counter('warn', 'missing gross', 1) 

    def reducer(self, key, values): 
     topten = [] 

     # For 'salary' and 'gross' compute the top 10 
     for p in values: 
      topten.append(p) 
      topten.sort() 
      topten = topten[-10:] 

     for p in topten: 
      yield key, p 

    combiner = reducer 

if __name__ == '__main__': 
salarymax.run() 

Answer 1

Ok.Sie müssen die Datei core-site.xml bearbeiten:

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
     <name>fs.default.name</name> 
     <value>hdfs://master:9000</value> 
    </property> 
</configuration> 

und die Datei hdfs-site.xml als:

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
     <name>dfs.replication</name> 
     <value>2</value> 
    </property> 
    <property> 
     <name>dfs.permissions</name> 
     <value>false</value> 
    </property> 
    <property> 
     <name>dfs.namenode.name.dir</name> 
     <value>/home/edureka/hadoop-2.7.3/namenode</value> 
    </property> 
    <property> 
     <name>dfs.datanode.data.dir</name> 
     <value>/home/edureka/hadoop-2.7.3/datanode</value> 
    </property> 
</configuration> 

und Sie müssen hdfs-site.xml bearbeiten:

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
     <name>dfs.replication</name> 
     <value>2</value> 
    </property> 
    <property> 
     <name>dfs.permissions</name> 
     <value>false</value> 
    </property> 
    <property> 
     <name>dfs.datanode.data.dir</name> 
     <value>/home/edureka/hadoop-2.7.3/datanode</value> 
    </property> 
</configuration> 

und Sie eine mapred-site.xml erstellen müssen Datei mit Inhalt:

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
     <name>mapreduce.framework.name</name> 
     <value>yarn</value> 
    </property> 
</configuration> 

und Sie müssen yarn-site.xml bearbeiten enthalten:

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
     <name>yarn.nodemanager.aux-services</name> 
     <value>mapreduce_shuffle</value> 
    </property> 
    <property> 
     <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name> 
     <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
    </property> 
</configuration> 

Dann tun:

start-dfs.sh 
start-yarn.sh 

dann tun:

hdfs dfs -mkdir /user/ 
hdfs dfs -mkdir /user/me/ 
hdfs dfs -mkdir /user/me/input/ 
hdfs dfs -put /home/me/Desktop/work/cv/hadoop/salaries.csv /user/me/input/ 

jetzt gerade:

sudo chmod a+x /home/me/Desktop/work/cv/hadoop/top_salaries.py 
python2 top_salaries.py -r hadoop hdfs:///user/me/input/salaries.csv > answer.csv 

funktioniert.