我正在尝试使用BroadcastState设置项目,但是由于某些原因,当我尝试运行该项目时出现此错误:
org.apache.flink.streaming.runtime.tasks.StreamTaskException:无法序列化操作员对象类org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperator。
我不确定为什么要扔它。传入和输出的对象(SampleInput和Token)是非常简单的avro生成的pojo,具有两个或三个字段,我尝试将BroadcastProcessFunction的方法留空以切出我可以设置的无法序列化的内容,但是仍然收到错误。这是代码的相关部分:
//Sideoutput that error strings will be written to
OutputTag<String> sideOutputTag = new OutputTag<String>("side-output") {};
//<Setup for broadcast state>
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.seconds(1))
.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
.cleanupFullSnapshot()
.build();
final MapStateDescriptor<String, Token> ruleStateDescriptor = new MapStateDescriptor<>(
"oathTokens",
BasicTypeInfo.STRING_TYPE_INFO,
AvroTypeInfo.of(new TypeHint<Token>() {}));
ruleStateDescriptor.enableTimeToLive(ttlConfig);
DataStream<Token> tokenObjectStream = tokenSourceStream.process(new JsonToTokenProcessFunction(sideOutputTag))
.startNewChain()
.uid("tokenObjectStream")
.name("tokenObjectStream");
BroadcastStream<Token> ruleBroadcastStream = tokenObjectStream.broadcast(ruleStateDescriptor);
//</Config for broadcast state>
//<Main Data Input Stream>
DataStream<SampleInput> jsonToSampleInput = kafkaStream.process(new JsonToPojoProcessFunction(sideOutputTag))
.startNewChain()
.uid("sampleInputStream")
.name("sampleInputStream");
BroadcastConnectedStream<SampleInput, Token> broadcastConnectedStream = jsonToSampleInput.connect(ruleBroadcastStream);
DataStream<SampleInput> matchedBroadcastStream = broadcastConnectedStream.process(new BroadcastProcessFunction<SampleInput, Token, SampleInput>() {
@Override
public void processElement(SampleInput sampleInput, ReadOnlyContext readOnlyContext, Collector<SampleInput> collector) throws Exception {
}
@Override
public void processBroadcastElement(Token token, Context context, Collector<SampleInput> collector) throws Exception {
}
});
任何帮助将不胜感激。我确定我只是忽略了一些东西。谢谢!
最佳答案
原来ttlConfig对象是无法序列化的。删除它可以解决问题。