spark streaming kafka Couldn't find leader

问题描述：

使用spark streaming接受kafka数据（使用direct方式）报错

Couldn't find leader offsets for Set([test,0], [test,1], [test,3])

 

 

Exception in thread "main" org.apache.spark.SparkException: java.nio.channels.ClosedChannelException

java.nio.channels.ClosedChannelException

org.apache.spark.SparkException: Couldn't find leader offsets for Set([test,0], [test,1], [test,3])

        at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)

        at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)

        at scala.util.Either.fold(Either.scala:97)

        at org.apache.spark.streaming.kafka.KafkaCluster$.checkErrors(KafkaCluster.scala:365)

        at org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:222)

        at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:484)

        at Apps.KafkaStreaming$.main(KafkaStreaming.scala:52)

        at Apps.KafkaStreaming.main(KafkaStreaming.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

解决办法：

尝试方法1. 因为我有3个broker 所以尝试更改 num.partitions=3  在创建一个新的 topic 仍旧不可以

尝试方法2.将本机host中维护服务器ip hostname，以及服务器中保证/etc/hosts下ip hostname关系  仍旧不可以

 

1.     停止kafka、zookeeper服务

2.     删除zookeeper、kafka   配置的logs下文件

3.     启动zookeeper、kafka服务

4.     创建新的topic

[hduser@BDTEST05 bin]$ ./kafka-console-producer.sh --broker-list  BDTEST05:9092 --topic test1

问题解决

注意 此处刚刚创建万的topic 如果立即跑程序  会报 是否存在这个topic 一定要topic里有内容了 才可以执行成功