Kafka之--多zookeeper，多broker之搭建
闲来如事，突发兴趣倒腾一下了kafka。其实类似环境搭建的博客很多，我这里算是整合一下，另外写出自己的过程与看法。完整的过程如下：
1）先准备好3台服务器。我准备的机器hostname/ip为：
kafka/192.168.56.151 , kafka2/192.168.56.152, kafka3/192.168.56.103
2) 搭建zookeeper集群。可以参考如下链接：
https://www.cnblogs.com/ysocean/p/9860529.html
搭建好，并配置好了环境变量后。可以分别启动3台机器，并查看集群状态。如下可以看到kafka机器为leader，kafka2 & kafka3为follower
[root@kafaka ~]# zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@kafaka ~]# zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: leader [root@kafaka3 ~]# zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@kafaka3 ~]# zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower [root@kafaka2 conf]# zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@kafaka2 conf]# zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower
3)安装配置kafka brokers。
kafka的安装过程，可以参考如下文档
https://blog.csdn.net/sunsijia21983/article/details/115305971
我在这里，配置了6个brokers。分别为broker0 ~ broker5。每台机器两个broker。
[root@kafaka config]# pwd /usr/local/kafka_2.13-2.7.0/config [root@kafaka config]# ls -l server.properties.* -rw-r--r-- 1 root root 7070 Jul 9 23:46 server.properties.0 -rw-r--r-- 1 root root 7055 Jul 10 00:06 server.properties.1
server.properties.0的配置如下：
注意需要的是每个broker有不同的id。而且端口不能相同。还有就是log.dirs的目录需要先创建。zookeeper.connect参数为所有3台服务器的zookeeper地址
在kafka2 & kafka3的机器需要使用不同的broker id，端口可以用同样的9092 & 9093，IP需要修改成机器IP(机器名也是可以的，前提是你配置了/etc/hosts文件
# The id of the broker. This must be set to a unique integer for each broker. broker.id=0 # Hostname and port the broker will advertise to producers and consumers. If not set, log.dirs=/usr/local/kafka_2.13-2.7.0/broker0 # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 #listeners=PLAINTEXT://192.168.56.150:9092 # Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if configured. Otherwise, it will use the value # returned from java.net.InetAddress.getCanonicalHostName(). #advertised.listeners=PLAINTEXT://your.host.name:9092 advertised.listeners=PLAINTEXT://192.168.56.151:9092 # The default number of log partitions per topic. More partitions allow greater # parallelism for consumption, but this will also result in more files across # the brokers. num.partitions=3 # Zookeeper connection string (see zookeeper docs for details). # This is a comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append an optional chroot string to the urls to specify the # root directory for all kafka znodes. zookeeper.connect=192.168.56.151:2181,192.168.56.152:2181,192.168.56.103:2181 server.properties.1的配置如下： broker.id=1 listeners=PLAINTEXT://192.168.56.151:9093 advertised.listeners=PLAINTEXT://192.168.56.151:9093 num.partitions=3 zookeeper.connect=192.168.56.151:2181,192.168.56.152:2181,192.168.56.103:2181
3)启动kafka(在kafka/bin目录下启动kafka进程）
kafka机器
[root@kafaka bin]# pwd /usr/local/kafka_2.13-2.7.0/bin 1018 ./kafka-server-start.sh ../config/server.properties.0 & 1019 ./kafka-server-start.sh ../config/server.properties.1 & kafka2机器 1025 ./kafka-server-start.sh ../config/server.properties.2 & 1026 ./kafka-server-start.sh ../config/server.properties.3 & kafka3机器 1003 ./kafka-server-start.sh ../config/server.properties.4 & 1004 ./kafka-server-start.sh ../config/server.properties.5 &
4)连接任一台机器的zookeeper，查看broker ids。如下可以看到我有0-5，六个broker上线了。
[root@kafaka bin]# zkCli.sh -server localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] ls / [admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper] [zk: localhost:2181(CONNECTED) 1] ls /brokers [ids, seqid, topics] [zk: localhost:2181(CONNECTED) 2] ls /brokers/ids [0, 1, 2, 3, 4, 5] [zk: localhost:2181(CONNECTED) 3]
如此，我可以算是搭建成功了一个高可用的kafka集群了。下一篇文章，我再来讲一讲如何测试kafka集群。
