filebeat+kafka失败

filebeat端配置

#----------------------------- Kafka output --------------------------------
output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["10.88.128.88:9092"]
  version: "0.8.2"

  topic: 'abc'
  #partition.round_robin:
  #  reachable_only: false

  required_acks: 1
  compression: none 
  #max_message_bytes: 1000000

 

报错如下：

2017/04/30 08:14:24.060269 log.go:12: WARN producer/broker/0 state change to [closing] because dial tcp: lookup test43 on 172.16.2.1:53: no such host
2017/04/30 08:14:24.060361 log.go:12: WARN producer/leader/rokid/0 state change to [retrying-48]
2017/04/30 08:14:24.060405 log.go:12: WARN producer/leader/rokid/0 abandoning broker 0
2017/04/30 08:14:24.060450 log.go:12: WARN producer/broker/0 shut down
x2017/04/30 08:14:24.160582 log.go:12: WARN client/metadata fetching metadata for [rokid] from broker 10.88.128.88:9092
2017/04/30 08:14:24.164350 log.go:12: WARN producer/broker/0 starting up
2017/04/30 08:14:24.164388 log.go:12: WARN producer/broker/0 state change to [open] on rokid/0
2017/04/30 08:14:24.164405 log.go:12: WARN producer/leader/rokid/0 selected broker 0
2017/04/30 08:14:24.164505 log.go:12: WARN producer/leader/rokid/0 state change to [flushing-48]
2017/04/30 08:14:24.164516 log.go:12: WARN producer/leader/rokid/0 state change to [normal]
i2017/04/30 08:14:24.185439 log.go:12: WARN Failed to connect to broker test43:9092: dial tcp: lookup test43 on 172.16.2.1:53: no such host
2017/04/30 08:14:24.185488 log.go:12: WARN producer/broker/0 state change to [closing] because dial tcp: lookup test43 on 172.16.2.1:53: no such host
2017/04/30 08:14:24.185534 log.go:12: WARN producer/leader/rokid/0 state change to [retrying-49]
2017/04/30 08:14:24.185552 log.go:12: WARN producer/leader/rokid/0 abandoning broker 0
2017/04/30 08:14:24.185592 log.go:12: WARN producer/broker/0 shut down
2017/04/30 08:14:24.285740 log.go:12: WARN client/metadata fetching metadata for [rokid] from broker 10.88.128.88:9092
2017/04/30 08:14:24.287170 log.go:12: WARN producer/broker/0 starting up
2017/04/30 08:14:24.287191 log.go:12: WARN producer/broker/0 state change to [open] on rokid/0
2017/04/30 08:14:24.287205 log.go:12: WARN producer/leader/rokid/0 selected broker 0
2017/04/30 08:14:24.287300 log.go:12: WARN producer/leader/rokid/0 state change to [flushing-49]
2017/04/30 08:14:24.287311 log.go:12: WARN producer/leader/rokid/0 state change to [normal]
2017/04/30 08:14:24.304521 log.go:12: WARN Failed to connect to broker test43:9092: dial tcp: lookup test43 on 172.16.2.1:53: no such host

上网查，filebeat解析的时候要解析到主机名，

所以绑定hosts如下

　　10.88.128.88 test43

问题搞定

按照报错，filebeat应该是先以配置的IP获取hostname，然后按照hostname连接kafka，如果通过hostname找不到对方服务器，则报错，所以需要在filebeat机器上绑定对方机器的hosts，暂不理解filebeat为什么要这么做。

如果目标机器中hosts文件书写有误，也会报错（filebeat向kafka中传递数据的时候还会检查目标机器的hosts文件）

 

开始还以为kafka版本太新，修改为filebeat目前支持的稳定版0.8.2都出问题，反复查，绑定hosts问题搞定。

 

参考网址：http://www.iyunv.com/thread-350295-1-1.html