Hadoop-Karte zu reduzieren hipi.mapreduce.BinaryOutputFormat nicht gefunden

Question

Wenn Sie versuchen, einen Job zu laufen Gesicht auf Bilder in einem HIB, bekomme ich diese Ausnahme. Ich habe hipi-2.1.0.jar in den Klassenpfad aufgenommen.

Exit code: 1 
Stack trace: ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) 
at org.apache.hadoop.util.Shell.run(Shell.java:487) 
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) 

Im Hadoop-Jobprotokoll wird der folgende Stacktrace gedruckt.

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.hipi.mapreduce.BinaryOutputFormat not found 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:519) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:499) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1598) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:499) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:285) 
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1556) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:422) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486) 
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.hipi.mapreduce.BinaryOutputFormat not found 
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) 
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) 
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:515) 
    ... 11 more 
Caused by: java.lang.ClassNotFoundException: Class org.hipi.mapreduce.BinaryOutputFormat not found 
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) 
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) 
    ... 13 more 

Wie wird das Problem gelöst?

Answer 1

Sieht aus, dass hadoop nicht das Klassenpfadverzeichnis verwendet, haben Sie versucht, hipi-2.1.0.jar in den Ordner hadoop-mapreduce/lib/zu platzieren?

Wenn Sie Cloudera verwenden, ist dies der Speicherort, an dem Sie die Bibliothek platzieren müssen, und sie sollte sich auf allen Knoten befinden.

/opt/cloudera/parzellen/CDH/lib/hadoop-mapreduce/lib/