Redshift Abfrage von EMR Spark-

Question

begann ich mit den Funken Shell auf Master-Knoten mit

spark-shell --jars minimal-json-0.9.4.jar spark-avro_2.11-3.1.0.jar spark-redshift_2.10-0.6.1-SNAPSHOT.jar 

Abfrage der Rotverschiebung Tabelle mit

val DF = sqlContext.read.format("com.databricks.spark.redshift").option("url", jdbcURL).option("tempdir", s3TempDir).option("query", myQuery).option("temporary_aws_access_key_id", awsAccessKey).option("temporary_aws_secret_access_key", awsSecretKey).option("temporary_aws_session_token", token).load() 

die Fehler:

java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.redshift. Please find packages at https://cwiki.apache.org/confluence/display/SPARK/Third+Party+Projects 
    at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:148) 
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:79) 
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:79) 
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340) 
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149) 
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:122) 
    ... 54 elided 
Caused by: java.lang.ClassNotFoundException: com.databricks.spark.redshift.DefaultSource 
    at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132) 
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132) 
    at scala.util.Try$.apply(Try.scala:192) 
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132) 
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132) 
    at scala.util.Try.orElse(Try.scala:84) 
    at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:132) 
    ... 59 more 

Answer 1

Sie haben vergessene Kommata:

spark-shell --jars minimal-json-0.9.4.jar,spark-avro_2.11-3.1.0.jar,spark-redshift_2.10-0.6.1-SNAPSHOT.jar