本文介绍了Spark Java IllegalArgumentException 在 org.apache.xbean.asm5.ClassReader的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我正在尝试将 Spark 2.3.1 与 Java 结合使用.
I'm trying to use Spark 2.3.1 with Java.
我遵循文档中的示例 但在调用 .fit(trainingData) 时不断得到描述不佳的异常.
I followed examples in the documentation but keep getting poorly described exception when calling .fit(trainingData).
Exception in thread "main" java.lang.IllegalArgumentException
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:46)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:449)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:432)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:103)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:432)
at org.apache.xbean.asm5.ClassReader.a(Unknown Source)
at org.apache.xbean.asm5.ClassReader.b(Unknown Source)
at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:262)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:261)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:261)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2299)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2073)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1358)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.take(RDD.scala:1331)
at org.apache.spark.ml.tree.impl.DecisionTreeMetadata$.buildMetadata(DecisionTreeMetadata.scala:112)
at org.apache.spark.ml.tree.impl.RandomForest$.run(RandomForest.scala:105)
at org.apache.spark.ml.classification.DecisionTreeClassifier.train(DecisionTreeClassifier.scala:116)
at org.apache.spark.ml.classification.DecisionTreeClassifier.train(DecisionTreeClassifier.scala:45)
at org.apache.spark.ml.Predictor.fit(Predictor.scala:118)
at com.example.spark.MyApp.main(MyApp.java:36)
我将这个虚拟数据集用于分类(data.csv):
I took this dummy dataset for classification (data.csv):
f,label
1,1
1.5,1
0,0
2,2
2.5,2
我的代码:
SparkSession spark = SparkSession.builder()
.master("local[1]")
.appName("My App")
.getOrCreate();
Dataset<Row> data = spark.read().format("csv")
.option("header", "true")
.option("inferSchema", "true")
.load("C:\\tmp\\data.csv");
data.show(); // see output(1) below
VectorAssembler assembler = new VectorAssembler()
.setInputCols(Collections.singletonList("f").toArray(new String[0]))
.setOutputCol("features");
Dataset<Row> trainingData = assembler.transform(data)
.select("features", "label");
trainingData.show(); // see output(2) below
DecisionTreeClassifier clf = new DecisionTreeClassifier();
DecisionTreeClassificationModel model = clf.fit(trainingData); // fails here (MyApp.java:36)
Dataset<Row> predictions = model.transform(trainingData);
predictions.show(); // never reached
输出(1):
+---+-----+
| f|label|
+---+-----+
|1.0| 1|
|1.5| 1|
|0.0| 0|
|2.0| 2|
|2.5| 2|
+---+-----+
输出(2):
+--------+-----+
|features|label|
+--------+-----+
| [1.0]| 1|
| [1.5]| 1|
| [0.0]| 0|
| [2.0]| 2|
| [2.5]| 2|
+--------+-----+
我的 build.gradle 文件如下所示:
My build.gradle file looks like this:
plugins {
id 'java'
id 'application'
}
group 'com.example'
version '1.0-SNAPSHOT'
sourceCompatibility = 1.8
mainClassName = 'MyApp'
repositories {
mavenCentral()
}
dependencies {
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.3.1'
compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.3.1'
compile group: 'org.apache.spark', name: 'spark-mllib_2.11', version: '2.3.1'
}
我错过了什么?
What am I missing?
推荐答案
您的机器上下载了哪个 Java 版本?您的问题可能与 Java 9 有关.
What Java version do you have downloaded on your machine?Your problem is probably related to Java 9.
如果您下载 Java 8(例如 jdk-8u171)，异常将消失，predictions.show() 的 output(3) 将如下所示:
If you download Java 8 (jdk-8u171, for instance), the Exception will disappear, and output(3) of predictions.show() will look like this:
+--------+-----+-------------+-------------+----------+
|features|label|rawPrediction| probability|prediction|
+--------+-----+-------------+-------------+----------+
| [1.0]| 1|[0.0,2.0,0.0]|[0.0,1.0,0.0]| 1.0|
| [1.5]| 1|[0.0,2.0,0.0]|[0.0,1.0,0.0]| 1.0|
| [0.0]| 0|[1.0,0.0,0.0]|[1.0,0.0,0.0]| 0.0|
| [2.0]| 2|[0.0,0.0,2.0]|[0.0,0.0,1.0]| 2.0|
| [2.5]| 2|[0.0,0.0,2.0]|[0.0,0.0,1.0]| 2.0|
+--------+-----+-------------+-------------+----------+
这篇关于Spark Java IllegalArgumentException 在 org.apache.xbean.asm5.ClassReader的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！