Spark Streaming 1.6.1不适用于Kinesis ASL 1.6.1和ASL 2.0.0-Preview

问题描述

我正在尝试使用Kinesis在EMR上运行火花流作业.带有Kinesis ASL 1.6.1的Spark 1.6.1写一个简单的单词计数示例.

I am trying to run spark streaming job on EMR with Kinesis. Spark 1.6.1 with Kinesis ASL 1.6.1. Writing a plain sample wordcount example.

        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kinesis-asl_2.10</artifactId>
        <version>1.6.1</version>
    </dependency>


    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>amazon-kinesis-client</artifactId>
        <version>1.6.3</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>amazon-kinesis-producer</artifactId>
        <version>0.10.2</version>
    </dependency>

这引发以下异常

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: com/google/protobuf/ProtocolStringList
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:157)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.consumeShard(ShardConsumer.java:126)

升级到2.0.0版预览

Upgrading to 2.0.0-preview

        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kinesis-asl_2.10</artifactId>
        <version>2.0.0-preview</version>
    </dependency>

给出以下例外情况

java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging

在org.apache.spark.streaming.kinesis.KinesisUtils $$ anonfun $ createStream $ 1.apply(KinesisUtils.scala:74)

at org.apache.spark.streaming.kinesis.KinesisUtils$$anonfun$createStream$1.apply(KinesisUtils.scala:74)

推荐答案

它是由protobuf-java依赖冲突引起的.使用mvn dependency:tree查找protobuf-java的版本，该版本是KCL和KPL所依赖的.并转到spark lib目录，您将找到另一个版本.请使用maven-shade-plugin，然后重新定位冲突类:

It was caused by protobuf-java dependency conflict. Use mvn dependency:tree to find the version of protobuf-java, which is KCL and KPL depend on. And go to spark lib directory, you would find the another version.Please use maven-shade-plugin, and relocate the conflict classes：

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <outputFile>
                    ${project.build.directory}/${project.artifactId}-${project.version}-selfcontained.jar
                </outputFile>
                <relocations>
                    <relocation>
                        <pattern>com.google.protobuf</pattern>
                        <shadedPattern>shade.com.google.protobuf</shadedPattern>
                    </relocation>
                    <relocation>
                        <pattern>com.amazonaws</pattern>
                        <shadedPattern>shade.com.amazonaws</shadedPattern>
                    </relocation>
                </relocations>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

这篇关于Spark Streaming 1.6.1不适用于Kinesis ASL 1.6.1和ASL 2.0.0-Preview的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！