kafka 集群安装
环境：
10.23.209.70 zookeep-kafka-node1 zookeep+kafka CentOS Linux release 7.1.1503
10.23.209.71 zookeep-kafka-node2 zookeep+kafka CentOS Linux release 7.1.1503
10.23.209.72 zookeep-kafka-node3 zookeep+kafka CentOS Linux release 7.1.1503
官方网站：
JDK下载地址：https://www.oracle.com/technetwork/java/javase/downloads/index.html
Zookeep官方下载地址：https://zookeeper.apache.org/releases.html
kafka官方下载地址：http://kafka.apache.org/downloads
前期准备关闭防火墙，selinux
[root@zookeep-kafka-node1 ~]# setenforce 0 [root@zookeep-kafka-node1 ~]# getenforce Permissive [root@zookeep-kafka-node1 ~]# cat /etc/selinux/config | grep 'SELINUX=' && -v ^# # SELINUX= can take one of these three values: SELINUX=disabled [root@zookeep-kafka-node1 ~]# systemctl stop firewalld [root@zookeep-kafka-node1 ~]# systemctl disable firewalld
一.JDK1.8安装
1.解压jdk至指定目录
[root@zookeep-kafka-node1 local]# tar zxvf jdk-8u231-linux-x64.tar.gz -C /usr/local/
2.修改vi /etc/profile 增加JDK环境变量
export JAVA_HOME=/usr/local/jdk1.8.0_231 export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
3.使环境变量生效
[root@zookeep-kafka-node1 local]# source /etc/profile
4.测试java环境，查看到java信息说明部署成功
[root@zookeep-kafka-node1 local]# java -version java version "1.8.0_231" Java(TM) SE Runtime Environment (build 1.8.0_231-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
5.将JDK，SCP到其他2台机器上。并修改/etc/profile文件使其生效 JDK部署完毕
[root@zookeep-kafka-node1 local]# scp -r /usr/local/jdk1.8.0_231/ root@10.23.215.71:/usr/local/ [root@zookeep-kafka-node1 local]# scp -r /usr/local/jdk1.8.0_231/ root@10.23.215.72:/usr/local/
二.部署zookeep集群
1.解压zookeep压缩包至指定文件夹
tar zxvf apache-zookeeper-3.5.6-bin.tar.gz -C /opt/ mv /opt/apache-zookeeper-3.5.6-bin/ zookeeper-3.5.6
2.创建zookeep配置文件zoo.cfg内容
[root@zookeep-kafka-node1 conf]# cat zoo_sample.cfg | grep -v ^$|grep -v ^# >zoo.cfg
[root@zookeep-kafka-node1 conf]# cat zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181
修改配置把数据目录和日志目录分开，并增加集群信息
[root@zookeep-kafka-node1 conf]# cat zoo.cfg
tickTime=10000
initLimit=5
syncLimit=2
dataDir /opt/zookeeper-3.5.6/data/zkdata
dataLogDir=/opt/zookeeper-3.5.6/data/zklogs
clientPort=2181
server.1=10.23.209.70:2888:3888
server.2=10.23.209.71:2888:3888
server.3=10.23.209.72:2888:3888
创建数据和日志目录
[root@zookeep-kafka-node1 opt]# mkdir -p /opt/zookeeper-3.5.6/data/{zkdata,zklogs}
将zookeep scp到其他2台服务器
[root@zookeep-kafka-node1 opt]# scp -r /opt/zookeeper-3.5.6 root@10.23.215.71:/opt
[root@zookeep-kafka-node1 opt]# scp -r /opt/zookeeper-3.5.6 root@10.23.215.72:/opt
3.创建myid文件
[root@zookeep-kafka-node1 opt]# echo "1" > /opt/zookeeper-3.5.6/data/zkdata/myid
[root@zookeep-kafka-node2 opt]# echo "2" > /opt/zookeeper-3.5.6/data/zkdata/myid
[root@zookeep-kafka-node3 opt]# echo "3" > /opt/zookeeper-3.5.6/data/zkdata/myid
4.启动zookeep服务 看到2个follower 1个leader 集群正常
[root@zookeep-kafka-node1 bin]# /opt/zookeeper-3.5.6/bin/zkServer.sh start ZooKeeper JMX enabled by default Using config: /opt/zookeeper-3.5.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@zookeep-kafka-node1 bin]# /opt/zookeeper-3.5.6/bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/zookeeper-3.5.6/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Mode: follower
三.安装kafka
1.解压至指定目录，修改配置
[root@zookeep-kafka-node1 ~]# tar zxvf kafka_2.12-2.3.0.tgz -C /opt/ [root@zookeep-kafka-node1 ~]# cd /opt/kafka_2.12-2.3.0/config/ [root@zookeep-kafka-node1 config]# cp server.properties server.properties.bak [root@zookeep-kafka-node1 config]# grep -v ^# server.properties.bak | grep -v ^$ >server.properties [root@zookeep-kafka-node1 config]# cat server.properties broker.id=0 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 修改数据存放路径 集群信息 等 [root@zookeep-kafka-node1 config]# cat server.properties broker.id=70 port=9092 host.name=10.23.209.70 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/opt/kafka_2.12-2.3.0/logs num.partitions=3 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 log.cleaner.enable=false zookeeper.connect=10.23.209.70:2181,10.23.209.71:2181,10.23.209.72:2181 zookeeper.connection.timeout.ms=1000000 zookeeper.session.timeout.ms=1000000 delete.topic.enable=true
2.将kafka目录scp到其他2台服务器 并修改文件中的broker.id host.name
[root@zookeep-kafka-node2 config]# cat server.properties broker.id=71 port=9092 host.name=10.23.209.71 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/opt/kafka_2.11-2.0.0/logs num.partitions=3 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 log.cleaner.enable=false zookeeper.connect=10.23.209.70:2181,10.23.209.71:2181,10.23.209.72:2181 zookeeper.connection.timeout.ms=1000000 zookeeper.session.timeout.ms=1000000 delete.topic.enable=true
[root@zookeep-kafka-node3 config]# cat server.properties broker.id=72 port=9092 host.name=10.23.209.72 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/opt/kafka_2.11-2.0.0/logs num.partitions=3 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 log.cleaner.enable=false zookeeper.connect=10.23.209.70:2181,10.23.209.71:2181,10.23.209.72:2181 zookeeper.connection.timeout.ms=1000000 zookeeper.session.timeout.ms=1000000 delete.topic.enable=true
启动kafka
[root@zookeep-kafka-node1 opt]# /opt/kafka_2.12-2.3.0/bin/kafka-server-start.sh /opt/kafka_2.12-2.3.0/config/server.properties
[root@zookeep-kafka-node2 opt]# /opt/kafka_2.12-2.3.0/bin/kafka-server-start.sh /opt/kafka_2.12-2.3.0/config/server.properties
[root@zookeep-kafka-node3 opt]# /opt/kafka_2.12-2.3.0/bin/kafka-server-start.sh /opt/kafka_2.12-2.3.0/config/server.properties
[2019-11-20 12:57:07,175] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser) [2019-11-20 12:57:07,175] INFO Kafka startTimeMs: 1574225827167 (org.apache.kafka.common.utils.AppInfoParser) [2019-11-20 12:57:07,200] INFO [KafkaServer id=70] started (kafka.server.KafkaServer) [2019-11-20 12:57:26,017] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser) [2019-11-20 12:57:26,017] INFO Kafka startTimeMs: 1574225846011 (org.apache.kafka.common.utils.AppInfoParser) [2019-11-20 12:57:26,019] INFO [KafkaServer id=71] started (kafka.server.KafkaServer) [2019-11-20 12:57:33,535] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser) [2019-11-20 12:57:33,535] INFO Kafka startTimeMs: 1574225853528 (org.apache.kafka.common.utils.AppInfoParser) [2019-11-20 12:57:33,537] INFO [KafkaServer id=72] started (kafka.server.KafkaServer)
