Storm KafkaSpout停止使用来自Kafka Topic的消息

本文介绍了Storm KafkaSpout停止使用来自Kafka Topic的消息的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述 29岁程序员，3月因学历无情被辞！ 我的问题是Storm KafkaSpout在一段时间后停止使用来自Kafka主题的消息。在storm中启用调试时，我得到如下日志文件：My problem is that Storm KafkaSpout stopped to consume messages from Kafka topic after a period of time. When debug is enabled in storm, I get the log file like this: 2016-07-05 03:58： 55.042 oasdexecutor [INFO] 处理收到的消息FOR -2 TUPLE ：source：__system：-1，stream：__tick，id：{}，[30]2016-07-05 03:58:55.042 o.a.s.d.executor [INFO] Processing received message FOR -2 TUPLE: source: __system:-1, stream: __tick, id: {}, [30] 2016-07-05 03：59：25.042 oasdexecutor [INFO] 处理收到的消息FOR -2 TUPLE ：sour ce：__system：-1，stream：__tick，id：{}，[30]2016-07-05 03:59:25.042 o.a.s.d.executor [INFO] Processing received message FOR -2 TUPLE: source: __system:-1, stream: __tick, id: {}, [30] 2016-07-05 03：59：25.946 oasdexecutor [INFO] 处理收到的消息FOR -2 TUPLE ：source：__system：-1，stream：__ metric_tick，id：{}，[60]2016-07-05 03:59:25.946 o.a.s.d.executor [INFO] Processing received message FOR -2 TUPLE: source: __system:-1, stream: __metrics_tick, id: {}, [60]我的测试拓扑非常简单，One KafkaSpout和另一个Counter Bolt。当拓扑工作正常时， FOR 和 TUPLE 之间的值为正数;当拓扑停止使用消息时，该值变为负数。所以我很好奇是什么原因造成处理收到的消息FOR -2 TUPLE 的问题，以及如何解决这个问题？My test topology is really simple, One KafkaSpout and another Counter Bolt. When the topology works fine, the value between FOR and TUPLE is a positive number; when the topology stops to consume the message, the value becomes negative. so I'm curious about what causes the problem of Processing received message FOR -2 TUPLE, and how to fix this problem?由方式，我的实验环境是：By the way, my experiment environment is:推荐答案随着来自stom邮件列表的帮助我能够调整KafkaSpout并解决问题。以下设置适用于我。With the help from the stom mail list I was able to tune KafkaSpout and resolve the issue. The following settings work for me.config.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 2048);config.put(Config.TOPOLOGY_BACKPRESSURE_ENABLE, false);config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384);config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384);我通过发送20k-50k批次进行测试，突发之间暂停1秒。每条消息都是2048字节。I tested by sending 20k-50k batches with 1sec pause between bursts. Each message was 2048 bytes.我正在运行3节点集群，我的拓扑有4个喷口，主题有64个分区。I am running 3 node cluster, my topology has 4 spouts and topic has 64 partitions.在200M消息之后它仍在工作....After 200M messages its still working.... 这篇关于Storm KafkaSpout停止使用来自Kafka Topic的消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！ 上岸，阿里云！