单机部署和配置Kafka集群
安装zookeeper 3.4.11单机集群
1.说明
需要说明的几个问题：1）kafka依赖zookeeper,请先安装zookeeper。2）zookeeper依赖java环境，请先安装jdk。3）zookeeper单机集群规划：3个节点，端口号分别是2181、2812、2813。4）zookeeper单机集群关键在于隔离各节点的端口、数据。
2.zookeeper单机集群安装
1）zookeeper安装目录
/opt/app/zookeeper，如果没有，就创建有关目录文件。
2）zookeeper集群节点规划
/opt/app/zookeeper下3个节点的子文件夹是：2181,2182,2183，各个节点的数据文件夹data，数据日志文件夹datalogs。
3）安装zookeeper集群
安装zookeeper具体步骤如下：
安装目录：/opt/app/zookeeper，如果没有，就创建有关目录文件,我的虚拟机没有，所以我需要创建。
mkdir /opt/app/zookeeper -p
下载并且解压zookeeper安装
[root@elk01 software]# pwd
/opt/software
[root@elk01 software]# ll
total 84128
-rw-r--r--. 1 root root 49475271 Nov 1 2017 kafka_2.11-1.0.0.tgz
-rw-r--r--. 1 root root 36668066 Nov 9 2017 zookeeper-3.4.11.tar.gz
[root@elk01 software]# tar -zxvf zookeeper-3.4.11.tar.gz -C /opt/app/
创建zookeeper有关文件夹结构
[root@elk01 app]# pwd
/opt/app
[root@elk01 app]# ll
total 4
drwxr-xr-x 2 root root 6 Mar 14 09:44 zookeeper
drwxr-xr-x 10 502 games 4096 Nov 2 2017 zookeeper-3.4.11
[root@elk01 app]# mkdir -p zookeeper/218{1,2,3}/{data,datalogs}
复制zookeeper程序文件到各个节点
[root@elk01 app]# pwd
/opt/app
You have new mail in /var/spool/mail/root
[root@elk01 app]# ll
total 4
drwxr-xr-x 5 root root 42 Mar 14 09:49 zookeeper
drwxr-xr-x 10 502 games 4096 Nov 2 2017 zookeeper-3.4.11
[root@elk01 app]# cp -r zookeeper-3.4.11 zookeeper/2181
[root@elk01 app]# cp -r zookeeper-3.4.11 zookeeper/2182
[root@elk01 app]# cp -r zookeeper-3.4.11 zookeeper/2183
原来的zookeeper文件夹可以删除
[root@elk01 app]# rm -rf zookeeper-3.4.11/
[root@elk01 app]# pwd
/opt/app
[root@elk01 app]# ll
total 0
drwxr-xr-x 5 root root 42 Mar 14 09:49 zookeeper
修改zookeeper zoo.cfg配置
zoo.cfg是zookeeper的配置文件，放在各个节点的zookeeper-3.4.11/conf/文件夹下。可以复制zoo_sample.cfg配置模板。
配置节点1：
[root@elk01 app]# cd zookeeper/2181/zookeeper-3.4.11/conf/
[root@elk01 conf]# cp zoo_sample.cfg zoo.cfg
[root@elk01 conf]# grep "^[a-z]" zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/app/zookeeper/2181/data
clientPort=2181
dataLogDir=/opt/app/zookeeper/2181/datalogs
server.1=10.96.211.209:2881:3881
server.2=10.96.211.209:2882:3882
server.3=10.96.211.209:2883:3883
同理，节点2和节点3也这样配置。注意修改clientPort、dataDir、dataLogDir。
配置节点2：
[root@elk01 conf]# pwd
/opt/app/zookeeper/2182/zookeeper-3.4.11/conf
[root@elk01 conf]# cp zoo_sample.cfg zoo.cfg
[root@elk01 conf]# pwd
/opt/app/zookeeper/2182/zookeeper-3.4.11/conf
[root@elk01 conf]# grep "^[a-z]" zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/app/zookeeper/2182/data
clientPort=2182
dataLogDir=/opt/app/zookeeper/2182/datalogs
server.1=10.96.211.209:2881:3881
server.2=10.96.211.209:2882:3882
server.3=10.96.211.209:2883:3883
配置节点3：
[root@elk01 conf]# pwd
/opt/app/zookeeper/2183/zookeeper-3.4.11/conf
[root@elk01 conf]# cp zoo_sample.cfg zoo.cfg
[root@elk01 conf]# grep "^[a-z]" zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/app/zookeeper/2183/data
clientPort=2183
dataLogDir=/opt/app/zookeeper/2183/datalogs
server.1=10.96.211.209:2881:3881
server.2=10.96.211.209:2882:3882
server.3=10.96.211.209:2883:3883
zookeeper配置说明：
clientPort：客户端连接端口
server.x：集群中的节点（包括自己），x对应myid的数字，2个端口分别用于集群选举通讯
myid修改配置
每个节点必须有myid配置文件，记录节点的唯一标识，必须放在dataDir文件夹下
id值必须与上面配置的server.x中的x对应。
[root@elk01 zookeeper]# pwd
/opt/app/zookeeper
[root@elk01 zookeeper]# ll
total 0
drwxr-xr-x 5 root root 58 Mar 14 09:59 2181
drwxr-xr-x 5 root root 58 Mar 14 09:59 2182
drwxr-xr-x 5 root root 58 Mar 14 09:59 2183
[root@elk01 zookeeper]# touch 2181/data/myid && echo "1" > 2181/data/myid
[root@elk01 zookeeper]# touch 2182/data/myid && echo "2" > 2182/data/myid
[root@elk01 zookeeper]# touch 2183/data/myid && echo "3" > 2183/data/myid
查看myid文件内容
[root@elk01 zookeeper]# more 218*/data/myid
::::::::::::::
2181/data/myid
::::::::::::::
1
::::::::::::::
2182/data/myid
::::::::::::::
2
::::::::::::::
2183/data/myid
::::::::::::::
3
如果配置了iptables防火墙，需要开启有关端口，我自己虚拟机防火墙关了，所以不需要配置。
4）启动zookeeper集群
启动各个节点zookeeper
进入各个节点启动zookeeper，是为了让运行日志zookeeper.out生成在各个节点目录下。
[root@elk01 2181]# pwd
/opt/app/zookeeper/2181
[root@elk01 2181]# ll
total 4
drwxr-xr-x 2 root root 18 Mar 14 10:21 data
drwxr-xr-x 2 root root 6 Mar 14 09:49 datalogs
drwxr-xr-x 10 root root 4096 Mar 14 09:59 zookeeper-3.4.11
[root@elk01 2181]# zookeeper-3.4.11/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@elk01 2181]# cd ../2182/
[root@elk01 2182]# zookeeper-3.4.11/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2182/zookeeper-3.4.11/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@elk01 2182]# cd ../2183
[root@elk01 2183]# zookeeper-3.4.11/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2183/zookeeper-3.4.11/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
5）查看zookeeper集群状态
节点1：
[root@elk01 zookeeper]# pwd
/opt/app/zookeeper
[root@elk01 zookeeper]# ll
total 0
drwxr-xr-x 5 root root 79 Mar 14 10:26 2181
drwxr-xr-x 5 root root 79 Mar 14 10:26 2182
drwxr-xr-x 5 root root 79 Mar 14 10:26 2183
[root@elk01 zookeeper]# 2181/zookeeper-3.4.11/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../conf/zoo.cfg
Mode: follower
节点2：
[root@elk01 zookeeper]# 2182/zookeeper-3.4.11/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2182/zookeeper-3.4.11/bin/../conf/zoo.cfg
Mode: leader
节点3：
[root@elk01 zookeeper]# 2183/zookeeper-3.4.11/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2183/zookeeper-3.4.11/bin/../conf/zoo.cfg
Mode: follower
如上，leader表示主节点，follower表示从节点。目前状态是1主2从，状态正常！
6）客户端连接zookeeper集群测试
如下表示连接正常，zookeeper单机集群搭建大功告成！
[root@elk01 zookeeper]# pwd
/opt/app/zookeeper
[root@elk01 zookeeper]# 2181/zookeeper-3.4.11/bin/zkCli.sh
Connecting to localhost:2181
2022-03-14 10:32:25,115 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
2022-03-14 10:32:25,117 [myid:] - INFO [main:Environment@100] - Client environment:host.name=elk01
2022-03-14 10:32:25,117 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_312
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Red Hat, Inc.
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../build/classes:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../build/lib/*.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../conf:
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-1160.el7.x86_64
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/app/zookeeper
2022-03-14 10:32:25,119 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
Welcome to ZooKeeper!
2022-03-14 10:32:25,132 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1035] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2022-03-14 10:32:25,179 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@877] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2022-03-14 10:32:25,208 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1000041f3ec0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
安装kafka 2.11-1.0.0单机集群
1.说明
需要说明几个问题:
2.Kafka单机集群安装
1）单机Kafka集群安装和配置
Kafka单机集群具体部署和配置步骤如下：
安装目录：/opt/app/kafka，如果没有，就创建。
mkdir /opt/app/kafka -p
[root@elk01 software]# pwd
/opt/software
下载并且解压kafka安装包
[root@elk01 software]# tar -zxvf kafka_2.11-1.0.0.tgz
创建kafka有关目录结构
[root@elk01 app]# pwd
/opt/app
[root@elk01 app]# ll
total 0
drwxr-xr-x 2 root root 6 Mar 14 11:05 kafka
drwxr-xr-x 5 root root 42 Mar 14 09:49 zookeeper
[root@elk01 app]# mkdir -p kafka/909{2,3,4}/logs
复制kafka安装程序文件到各个节点目录下
[root@elk01 software]# pwd
/opt/software
[root@elk01 software]# ll
total 84128
drwxr-xr-x 6 root root 89 Oct 28 2017 kafka_2.11-1.0.0
-rw-r--r--. 1 root root 49475271 Nov 1 2017 kafka_2.11-1.0.0.tgz
-rw-r--r--. 1 root root 36668066 Nov 9 2017 zookeeper-3.4.11.tar.gz
[root@elk01 software]# cp -r kafka_2.11-1.0.0 /opt/app/kafka/9092/
[root@elk01 software]# cp -r kafka_2.11-1.0.0 /opt/app/kafka/9093/
You have new mail in /var/spool/mail/root
[root@elk01 software]# cp -r kafka_2.11-1.0.0 /opt/app/kafka/9094/
修改kafka集群配置
修改每个节点下kafka_2.11-1.0.0/config/server.properties配置文件。
需要修改的配置项修改后如下：
节点1：
[root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# vim 9092/kafka_2.11-1.0.0/config/server.properties
broker.id=1
listeners=PLAINTEXT:10.96.211.209:9092
advertised.listeners=PLAINTEXT://10.96.211.209:9092
log.dirs=/opt/app/kafka/9092/logs
zookeeper.connect=10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183
节点2：
root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# vim 9093/kafka_2.11-1.0.0/config/server.properties
broker.id=2
listeners=PLAINTEXT:10.96.211.209:9093
advertised.listeners=PLAINTEXT://10.96.211.209:9093
log.dirs=/opt/app/kafka/9093/logs
zookeeper.connect=10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183
节点3：
root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# vim 9094/kafka_2.11-1.0.0/config/server.properties
broker.id=3
listeners=PLAINTEXT://10.96.211.209:9094
advertised.listeners=PLAINTEXT://10.96.211.209:9094
log.dirs=/opt/app/kafka/9094/logs
zookeeper.connect=10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183
kafka consumer.properties配置文件修改
每个节点的consumer.properties只需要修改如下配置，需要修改的配置项修改后如下：
[root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# ll
total 0
drwxr-xr-x 4 root root 42 Mar 14 11:10 9092
drwxr-xr-x 4 root root 42 Mar 14 11:10 9093
drwxr-xr-x 4 root root 42 Mar 14 11:10 9094
vim 9092/kafka_2.11-1.0.0/config/consumer.properties
bootstrap.servers=10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094
vim 9093/kafka_2.11-1.0.0/config/consumer.properties
bootstrap.servers=10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094
vim 9094/kafka_2.11-1.0.0/config/consumer.properties
bootstrap.servers=10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094
如果开启了防火墙，需要开启有关端口，我的虚拟机防火墙关了，所以不需要配置。
2）启动Kafka集群
节点1：
[root@elk01 bin]# pwd
/opt/app/kafka/9092/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
节点2：
[root@elk01 bin]# cd /opt/app/kafka/9093/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
节点3：
[root@elk01 bin]# cd /opt/app/kafka/9094/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
3）测试Kafka集群是否可用
1）创建一个test主题（topic）
[root@elk01 bin]# pwd
/opt/app/kafka/9092/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-topics.sh --create --zookeeper 10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183 --replication-factor 3 --partitions 3 --topic 0314rontest
Created topic "0314rontest".
2）连接生产者，并发送数据
[root@elk01 bin]# pwd
/opt/app/kafka/9092/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-console-producer.sh --broker-list 10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094 --topic 0314rontest
>nihao
>haha
>hehe
>
3)新开一个ssh会话，连接消费者
如上，消费者收到消息，说明kafka集群是OK的！测试成功，kafka集群搭建大功告成！
