Flume常见问题

问题一：
ERROR [PollableSourceRunner-KafkaSource-r2] (org.apache.flume.source.kafka.KafkaSource.doProcess:314) - KafkaSource EXCEPTION, {}
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance
这个错误提示比较直白，意思是消费者消费了数据，但在规定时间内没有commit，所以kafka认为这个consumer挂掉了，这时对consumer的group进行再平衡。

增加消费超时时间。
消费超时时间通过heartbeat.interval.ms设置，heartbeat.interval.ms的大小不能超过session.timeout.ms。
session.timeout.ms必须在[group.min.session.timeout.ms, group.max.session.timeout.ms]范围内 

 

问题二：

ERROR [PollableSourceRunner-KafkaSource-r2] (org.apache.flume.source.kafka.KafkaSource.doProcess:314) - KafkaSource EXCEPTION, {}
org.apache.flume.ChannelFullException: The channel has reached it's capacity. This might be the result of a sink on the channel having too low of batch size, a downstream system running slower than normal, or t
hat the channel capacity is just too low. [channel=c3] 

 

问题三： 

ERROR [hdfs-k3-call-runner-1] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync:268) - Error while trying to hflushOrSync!
[SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:443) - HDFS IO error
java.io.IOException: Callable timed out after 10000 ms on file: hdfs://mycluster/test_flume/FlumeData.1597876537632.tmp 

 

问题四：
在file channel堆积数据过多，磁盘爆满，channel中的data数据积压过多
注意相关参数的设置（最好倍数关系）：
batchsize <=transactionCapacity<=capacity  

 

 

问题五：
oom问题解决：
在flume/bin/flume_ng文件中的 JAVA_OPTS="-Xmx20m"
改配置文件flume/conf/flume-env.sh export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote" 参数

 

问题六：

?

10 十月 2022 16:08:22,668 WARN  [hdfs-group-event-call-runner-11] (org.apache.hadoop.hdfs.DFSUtil.getAddressesForNameserviceId:688)  - Namenode for babeltimecluster remains unresolved for ID nn1.  Check your hdfs-site.xml file to ensure namenodes are configured properly.

10 十月 2022 16:08:22,669 WARN  [hdfs-group-event-call-runner-11] (org.apache.hadoop.hdfs.DFSUtil.getAddressesForNameserviceId:688)  - Namenode for babeltimecluster remains unresolved for ID nn2.  Check your hdfs-site.xml file to ensure namenodes are configured properly.

10 十月 2022 16:08:22,669 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:447)  - process failed

java.lang.IllegalArgumentException: java.net.UnknownHostException: yz-tpl-hadoop-xxx-xx

　　解决方式：把hadoop对应的hdfs-site.xml和core-site.xml 文件copy到 $flume/conf目录下；

对应jar包copy到lib下：