在Apache NiFi中使用AvroSchemaRegistry

问题描述

我有5种不同的 CSVReader 控制器服务.除了 schema text (因为标题不同)和1个 CSVRecordSetWriter 之外，它们的配置相同.

I have 5 different CSVReader controller services. Their configuration is the same except schema text (because different headers) and 1 CSVRecordSetWriter.

我只想保留一个 CSVReader 并动态设置 schema文本.我读到有关 AvroSchemaRegistry 的信息，但我不清楚如何使用它.

I want to leave only one CSVReader and set schema text dynamically. I read about AvroSchemaRegistry but I didn't clearly understand how to use it.

我是否应该创建5个具有2个属性的 AvroSchemaRegistry 控制器: name 和 value ?F.e.我想放入以下架构:

Should i create 5 different AvroSchemaRegistry controllers with 2 attributes: name and value?F.e. i want to put following schema:

{
    "type": "record",
    "name": "campaigns",
    "namespace": "common",
    "fields": [
        {"name": "campaign_name", "type": "string"},
        {"name": "campaign_id", "type": "long"},
        {"name": "date", "type" : {"type": "int", "logicalType" : "date"}}
    ]
}

我应该使用以下内容创建 AvroSchemaRegistry :

I should create AvroSchemaRegistry with:

• 名称 =广告系列；
• value = 上面的完整代码.

• name = campaigns;
• value = Full Code above.

对于另一个模式，我应该创建另一个具有其他属性 name 和 value ?

的 AvroSchemaResgitry 控制器?

For another schema i should create another AvroSchemaResgitry controller with another attributes name and value?

之后，如何配置 CSVReader 和 CSVRecordSetWriter 以使用这些架构?最后，我应该如何处理流文件?添加其他属性?什么样的?

After that, how to configure CSVReader and CSVRecordSetWriter to work with these schemas? And finally, what should i do with flowfiles? Add additional attributes? What kind of?

推荐答案

以这种方式配置您的流程(根据您的要求进行更改)，

Configure your flow something like this (make changes as per your requirement) ,

1. UpdateAttribute 配置以派生/硬编码流文件特定的模式-

1. UpdateAttribute configuration to derive/hard code flowfile specific schema-

1. ValidateRecord 配置以使用通用的csv阅读器并动态传递模式-

1. ValidateRecord configuration to use generic csv reader and pass schema dynamically -

1. CSVReader 控制器服务以使用动态传递的架构并设置架构访问策略-

1. CSVReader controller service to use dynamically passed schema and set schema access strategy -

如果您想使用NiFi支持的架构注册表，则将所有架构放入注册表中，并设置 schema.name 和访问策略属性，以供记录读取器/写入器访问注册表中创建模式，但首先需要在控制器服务中添加/配置模式注册表提供程序.

If you wish to use NiFi supported schema registry then put all of your schemas in registry and set schema.name and access strategy properties for record reader/writer to access schema from registry, but first you need to add/configure schema registry provider in controller service.

这篇关于在Apache NiFi中使用AvroSchemaRegistry的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！