kafka安装
下载最近的kafka包,下载地址为
http://mirrors.hust.edu.cn/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz
解压:
tar -xzf kafka_2.11-2.1.0.tgz
前提需启动zk
启动kafka
cd kafka_2.11-2.1.0 bin/zookeeper-server-start.sh config/zookeeper.properties
查看topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
创建topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
执行命令后显示
Created topic "test".
再次查看topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
返回结果
test
启动生产者,并发送消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >this is message >this is message too. >
启动消费者接收消息
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning this is message this is message too.
在部署集群过程中会遇到以下问题
[2019-01-02 14:41:03,450] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:264) at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693) at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348) at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372) at kafka.server.KafkaServer.startup(KafkaServer.scala:202) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) at kafka.Kafka$.main(Kafka.scala:75) at kafka.Kafka.main(Kafka.scala)
这个异常时好时坏,具体情况是连接本机的zk地址是好使的,连接其他服务器的zk就有问题,解决办法是配置/etc/hosts
10.6.124.17 test_124_17 10.6.124.18 test_124_18 10.6.124.19 test_124_19
集群中启动生产者
bin/kafka-console-producer.sh --broker-list 10.6.124.17:9092 --topic test
集群中启动消费者
bin/kafka-console-consumer.sh --bootstrap-server 10.6.124.18:9092 --topic test --from-beginning