NoSuchMethodError: org.apache.spark.sql.SQLContext.applySchema

问题描述

我正在尝试使用 Apache Spark 中提供的 sqlcontext 使用以下代码查询存储在 hdfs 中的文件，但出现 NoSuchMethodError

打包SQL导入 org.apache.spark.SparkContext导入 org.apache.spark.sql._对象 SparSQLCSV { def main(args: Array[String]) {val sc = new SparkContext("local[*]","home")val sqlContext = 新的 org.apache.spark.sql.SQLContext(sc)val people = sc.textFile("/home/devan/Documents/dataset/peoplesTest.csv")val 分隔符 = ","val schemaString = "a,b".split(delimiter)//csv header//自动模式创建val schema = StructType(schemaString.map(fieldName => StructField(fieldName, StringType, true)))val peopleLines = people.flatMap(x=> x.split("\n"))val rowRDD = peopleLines.map(p=>{Row.fromSeq(p.split(delimiter))})val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)peopleSchemaRDD.registerTempTable("people")sqlContext.sql("SELECT b FROM people").foreach(println)} }

线程main"中的异常java.lang.NoSuchMethodError:org.apache.spark.sql.SQLContext.applySchema(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/DataFrame;在 scalding.Main_Obj$.main(Main_Obj.scala:34)在 scalding.Main_Obj.main(Main_Obj.scala)在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在 java.lang.reflect.Method.invoke(Method.java:606)在 org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)在 org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)在 org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我已经使用 spark 中提供的命令行尝试了相同的方法，它可以工作，但是当我创建一个 Scala 项目并尝试运行它时，我收到了上述错误.我做错了什么?

解决方案

NoSuchMethodError 通常意味着您在库之间存在不兼容.在这种特殊情况下，您可能正在使用需要 Spark 1.3 和旧版 Spark 的 spark-csv 版本.

I am trying to query on a file stored in hdfs using sqlcontext provided in Apache Spark using the below code but i am getting a NoSuchMethodError

package SQL

import org.apache.spark.SparkContext 
import org.apache.spark.sql._

object SparSQLCSV {   def main(args: Array[String]) {

    val sc = new SparkContext("local[*]","home")
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    val people = sc.textFile("/home/devan/Documents/dataset/peoplesTest.csv")
    val delimiter = ","
    val schemaString = "a,b".split(delimiter)//csv header
    //Automated Schema creation
    val schema =   StructType(schemaString.map(fieldName => StructField(fieldName, StringType, true)))
    val peopleLines = people.flatMap(x=> x.split("\n"))
    val rowRDD = peopleLines.map(p=>{
      Row.fromSeq(p.split(delimiter))
    })
    val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)
    peopleSchemaRDD.registerTempTable("people")
    sqlContext.sql("SELECT b FROM people").foreach(println)

  } }

I have tried the same using command line provided in spark, and it works but when i create a scala project and try to run it i get the above error. what am i doing wrong ?

解决方案

NoSuchMethodError usually means that you have incompatibles between libraries. In this particular case it looks like you might be using a version of spark-csv that requires Spark 1.3 with an older version of Spark.

这篇关于NoSuchMethodError: org.apache.spark.sql.SQLContext.applySchema的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！