问题描述
我正在尝试运行小型 Spark 应用程序，但出现以下异常:
I'm trying to run small spark application and am getting the following exception:
Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:262)
at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:217)
at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
相关的gradle依赖部分:
the relevant gradle dependencies section:
compile('org.apache.spark:spark-core_2.10:1.3.1')
compile('org.apache.hadoop:hadoop-mapreduce-client-core:2.6.2') {force = true}
compile('org.apache.hadoop:hadoop-mapreduce-client-app:2.6.2') {force = true}
compile('org.apache.hadoop:hadoop-mapreduce-client-shuffle:2.6.2') {force = true}
compile('com.google.guava:guava:19.0') { force = true }
推荐答案
version 2.6.2 of hadoop:hadoop-mapreduce-client-core 无法使用连同 guava 的新版本(我尝试了 17.0 - 19.0)，因为 guava 的 StopWatch 构造函数无法访问(导致上面的IllegalAccessError)
version 2.6.2 of hadoop:hadoop-mapreduce-client-core can't be used together with guava's new versions (I tried 17.0 - 19.0) since guava's StopWatch constructor can't be accessed (causing above IllegalAccessError)
使用 hadoop-mapreduce-client-core 的最新版本 - 2.7.2(其中他们不使用 guava's StopWatch 在上面的方法中，而不是他们使用 org.apache.hadoop.util.StopWatch) 解决了这个问题，需要两个额外的依赖:
using hadoop-mapreduce-client-core's latest version - 2.7.2 (in which they don't use guava's StopWatch in the above method, rather they use org.apache.hadoop.util.StopWatch) solved the problem, with two additional dependencies that were required:
compile('org.apache.hadoop:hadoop-mapreduce-client-core:2.7.2') {force = true}
compile('org.apache.hadoop:hadoop-common:2.7.2') {force = true} // required for org.apache.hadoop.util.StopWatch
compile('commons-io:commons-io:2.4') {force = true} // required for org.apache.commons.io.Charsets that is used internally
注意:有两个 org.apache.commons.io 包:commons-io:commons-io(我们在这里)，以及org.apache.commons:commons-io(旧版，2007 年).确保包含正确的.
note:there are two org.apache.commons.io packages:commons-io:commons-io (ours here), andorg.apache.commons:commons-io (old one, 2007). make sure to include the correct one.
这篇关于从 org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus 到 guava 的 StopWatch 的 IllegalAccessError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！