kafka安装

下载最近的kafka包，下载地址为

http://mirrors.hust.edu.cn/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz 

解压：

tar -xzf kafka_2.11-2.1.0.tgz

前提需启动zk

启动kafka

cd kafka_2.11-2.1.0
bin/zookeeper-server-start.sh config/zookeeper.properties

查看topic

bin/kafka-topics.sh --list --zookeeper localhost:2181

创建topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

执行命令后显示

Created topic "test".

再次查看topic 

bin/kafka-topics.sh --list --zookeeper localhost:2181

返回结果

test

启动生产者，并发送消息

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>this is message
>this is message too.
>

启动消费者接收消息

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
this is message
this is message too.

在部署集群过程中会遇到以下问题

[2019-01-02 14:41:03,450] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
    at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
    at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
    at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:264)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
    at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:75)
    at kafka.Kafka.main(Kafka.scala)

这个异常时好时坏，具体情况是连接本机的zk地址是好使的，连接其他服务器的zk就有问题，解决办法是配置/etc/hosts

10.6.124.17 test_124_17
10.6.124.18 test_124_18
10.6.124.19 test_124_19

 集群中启动生产者

bin/kafka-console-producer.sh --broker-list 10.6.124.17:9092 --topic test

集群中启动消费者

bin/kafka-console-consumer.sh --bootstrap-server 10.6.124.18:9092 --topic test --from-beginning