Kafka Connect(单消息转换)行过滤

问题描述

我阅读了有关Kafka 0.10.2.1中引入的Kafka connect转换的信息 https://kafka.apache.org/documentation/#connect_transforms

I read about Kafka connect transformations introduced in kafka 0.10.2.1https://kafka.apache.org/documentation/#connect_transforms

我注意到所有转换都是基于列的转换.我有一个用例，需要基于值的筛选.例如:

I noticed that all the transformations are column based transformations.I have a use-case where I need value based filtering. For example:

请考虑以下一组人的数据集:

consider the following dataset of a group of people:

{"firstName": "FirstName1", "lastName": "LastName1", "age": 30}
{"firstName": "FirstName2", "lastName": "LastName2", "age": 30}
{"firstName": "FirstName3", "lastName": "LastName1", "age": 60}
{"firstName": "FirstName4", "lastName": "LastName2", "age": 60}

我希望我的工作人员过滤所有姓氏为LastName2的记录

I want my worker to filter all those records whose lastName is LastName2

是否可以使用kafka-connect或为此用例编写一个单独的程序.

Is it possible using kafka-connect or I need to write a separate program for this use-case.

谢谢

推荐答案

没有理由您无法使用单个消息转换"解决此问题-但由于您所描述的内容不可用，因此需要编写自定义的通过当前交付的转换.

No reason why you couldn't solve this with Single Message Transforms - but you'd need to write a custom one since what you're describing is not available through the transforms currently shipped.

这是关于何时使用和不使用SMT的有用讨论: 2017年纽约卡夫卡峰会:单消息转换不是您想要的转换(Ewen Cheslack-Postava，合流)

This is a useful talk here on when to use, and not to use, SMTs: Kafka Summit New York 2017 : Single Message Transformations Are Not the Transformations You’re Looking For (Ewen Cheslack-Postava, Confluent)

编辑2020年4月:现在有一个过滤器SMT 作为Confluent平台的一部分提供

Edit April 2020: There is now a Filter SMT provided as part of Confluent Platform

这篇关于Kafka Connect(单消息转换)行过滤的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！