Funken hat kein Dateisystem

Question

Es war Arbeit auf lokaler, aber nicht auf Linux-Knoten arbeiten. Ich habe ein Fettglas mit sbt-Assembly gemacht.

hier sind meine plugins.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.1") 
addSbtPlugin("org.xerial.sbt" % "sbt-pack" % "0.7.7") 

Ich habe versucht, CSV-Datei von databricks.spark Paket, aber keine Verwendung hdfs zu laden. hier ist Teil, wo ich denke, Fehler kommt auf.

val logFile = "/spark/data/test.csv" 
val rawdf = sqlContext.read 
    .format("com.databricks.spark.csv") 
    .option("header", "true") // Use first line of all files as header 
    .option("inferSchema", "true") // Automatically infer data types 
    .load(logFile) 

aber irgendwie funktioniert es nicht. hier ist Lognachricht

16/03/22 18:36:24 INFO MemoryStore: ensureFreeSpace(213512) called with curMem=242504, maxMem=555         755765 
16/03/22 18:36:24 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size         208.5 KB, free 529.6 MB) 
16/03/22 18:36:24 INFO MemoryStore: ensureFreeSpace(19788) called with curMem=456016, maxMem=5557         55765 
16/03/22 18:36:24 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated         size 19.3 KB, free 529.6 MB) 
16/03/22 18:36:24 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.2.18:42276         (size: 19.3 KB, free: 530.0 MB) 
16/03/22 18:36:24 INFO SparkContext: Created broadcast 4 from textFile at TextFile.scala:30 
16/03/22 18:36:24 INFO FileInputFormat: Total input paths to process : 1 
Exception in thread "main" java.io.IOException: No FileSystem for scheme: d 
     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) 
     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) 
     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) 
     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) 
     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) 
     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) 
     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) 
     at org.apache.spark.SparkHadoopWriter$.createPathFromString(SparkHadoopWriter.scala:170) 
     at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDD         Functions.scala:988) 
     at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctio         ns.scala:965) 
     at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctio         ns.scala:965) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) 
     at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) 
     at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:965) 
     at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDD         Functions.scala:897) 
     at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctio         ns.scala:897) 
     at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctio         ns.scala:897) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) 
     at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) 
     at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:896) 
     at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1430) 
     at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1409) 
     at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1409) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) 
     at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) 
     at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1409) 
     at destiny.spark.PACase.make(PACase.scala:58) 
     at TestPaApp$.main(TestPaApp.scala:114) 
     at TestPaApp.main(TestPaApp.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:497) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(Spar         kSubmit.scala:674) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
16/03/22 18:36:24 INFO SparkContext: Invoking stop() from shutdown hook 
16/03/22 18:36:24 INFO SparkUI: Stopped Spark web UI at http://192.168.2.18:4040 
16/03/22 18:36:24 INFO DAGScheduler: Stopping DAGScheduler 
16/03/22 18:36:24 INFO SparkDeploySchedulerBackend: Shutting down all executors 
16/03/22 18:36:24 INFO SparkDeploySchedulerBackend: Asking each executor to shut down 
16/03/22 18:36:24 INFO Master: Received unregister request from application app-20160322183602-00         14 
16/03/22 18:36:24 INFO Master: Removing app app-20160322183602-0014 
16/03/22 18:36:24 INFO Worker: Asked to kill executor app-20160322183602-0014/0 
16/03/22 18:36:24 INFO ExecutorRunner: Runner thread for executor app-20160322183602-0014/0 inter         rupted 

Answer 1

Sieht aus wie Sie eine Datei namens D:\... zu laden sind versuchen - solche Wege existieren nicht auf Linux, so dass die D: als Filesystem Schema analysiert wird (z hdfs:, file:). Ändern Sie den Dateinamen in etwas, auf das über den Linux-Knoten zugegriffen werden kann (lokale Linux-Datei oder ein Pfad auf einem freigegebenen Dateisystem wie HDFS).