Kafka集群部署
1：环境
kafka_node1 10.0.0.10 2C2G
kafka_node2 10.0.0.11 2C2G
kafka_node3 10.0.0.12 2C2G
2：基础配置(全部节点操作)
1：配置hostname、
[root@virtual_host ~]# hostnamectl set-hostname kafka-node1
[root@virtual_host ~]# hostnamectl set-hostname kafka-node2
[root@virtual_host ~]# hostnamectl set-hostname kafka-node3
2：写hosts
[root@kafka-node1 ~]# cat << eof>> /etc/hosts
10.0.0.10 kafka-node1
10.0.0.11 kafka-node2
10.0.0.12 kafka-node3
eof
3：免密配置
[root@kafka-node3 ~]# ssh-keygen
[root@kafka-node3 ~]# ssh-copy-id root@10.0.0.10
[root@kafka-node3 ~]# ssh-copy-id root@10.0.0.11
[root@kafka-node3 ~]# ssh-copy-id root@10.0.0.12
4：依赖准备
[root@kafka-node1 ~]# yum install -y java-1.8.0 java-1.8.0-openjdk-devel
3：配置Kafka和Zookeeper
1：下载 kafka
[root@kafka-node1 ~]# wget https://mirrors.cnnic.cn/apache/kafka/2.8.1/kafka_2.13-2.8.1.tgz --no-check-certificate
2：配置集群
解压
[root@kafka-node1 ~]# tar xf kafka_2.13-2.8.1.tgz
[root@kafka-node1 ~]# mv kafka_2.13-2.8.1 kafka
[root@kafka-node1 ~]# mv kafka /usr/local/
3：配置 zookeeper 集群
要搭建 kafka 集群，首先要部署 zookeeper 集群，这里直接使用 kafka 内置的 zookeeper 进行部署
[root@kafka-node1 config]# egrep -v '^#|^$' zookeeper.properties
dataDir=/usr/local/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
initLimit=10
syncLimit=5
server.1=kafka-node1:2888:3888
server.2=kafka-node2:2889:3889
server.3=kafka-node3:2890:3890
dataDir：zookeeper 的快照存储地址
clientPort：zookeeper 端口
maxClientCnxns：单个客户端与单台服务器之间的连接数的限制
initLimit：默认值是 10，即 tickTime 属性值的 10 倍。它用于配置允许 Followers 连接并同步到 Leader 的最大时间。如果 ZooKeeper 管理的数据量很大的话可以增加这个值
syncLimit：默认值是 5，即 tickTime 属性值的 5 倍。它用于配置 Leader 和 Followers 间进行心跳检测的最大延迟时间。如果在设置的时间内 Followers 无法与 Leader 进行通信，那么 Followers 将会被丢弃
server.1：server.x 中的 x 要与 myid 中的数字一致，node1 为集群 IP，当然前边配置了 hosts 可以这么写，2888 用于 follower 与 leader 之间的数据同步与其他通信；3888 用于 leader 选举时的通信
4：创建对应的 myid，用于区分不同节点的身份
[root@kafka-node1 config]# mkdir /usr/local/kafka/zookeeper
[root@kafka-node1 config]# echo "1" >>/usr/local/kafka/zookeeper/myid
[root@kafka-node2 config]# mkdir /usr/local/kafka/zookeeper
[root@kafka-node2 config]# echo "2" >>/usr/local/kafka/zookeeper/myid
[root@kafka-node3 config]# mkdir /usr/local/kafka/zookeeper
[root@kafka-node3 config]# echo "3" >>/usr/local/kafka/zookeeper/myid
5：启动 zookeeper 服务
[root@kafka-node1 config]# /usr/local/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.properties
[root@kafka-node2 config]# /usr/local/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.propertie
[root@kafka-node3 config]# /usr/local/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.propertie
或者配置systemd管理
[root@kafka-node1 ~]# cat << eof>>/lib/systemd/system/zookeeper.service
[Unit]
Description=zookeeper
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.properties
Restart=on-failure
[Install]
WantedBy=multi-user.target
eof
6：查看zk状态
[root@kafka-node1 ~]# ss -lnt
[root@kafka-node1 ~]# netstat -nplt | grep java | awk '{print $4,$7}'
:::46812 19710/java
:::2181 19710/java
10.0.0.10:3888 19710/java
:::8080 19710/java
# 验证
[root@kafka-node1 ~]# jps -m
19762 Jps -m
19710 QuorumPeerMain /usr/local/kafka/config/zookeeper.properties
7：配置 kafka 集群
[root@kafka-node1 ~]# cat << eof>/usr/local/kafka/config/server.properties
broker.id=1 #三个节点不能相同
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/local/kafka/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.0.10:2181,10.0.0.11:2181,10.0.0.12:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
eof
8：启动 kafka
[root@kafka-node1 ~]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
[root@kafka-node2 ~]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
[root@kafka-node3 ~]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
或者写systemd管理
[root@kafka-node1 ~]# cat << eof>>/lib/systemd/system/kafka.service
[Unit]
Description=kafka
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
Restart=on-failure
[Install]
WantedBy=multi-user.target
eof
# 验证
[root@kafka-node1 ~]# jps -m
20314 Jps -m
19710 QuorumPeerMain /usr/local/kafka/config/zookeeper.properties
20239 Kafka /usr/local/kafka/config/server.properties
4：Kafka集群使用
1：创建topic验证
[root@kafka-node1 ~]# /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 10.0.0.10:2181,10.0.0.11:2181,10.0.0.12:2181 --replication-factor 3 --partitions 1 --topic devops-topic
Created topic devops-topic.
2：查看topic
[root@kafka-node1 ~]# /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 10.0.0.10:2181,10.0.0.11:2181,10.0.0.12:2181 devopsdu-topic
devops-topic
3：生产消息
[root@kafka-node1 ~]# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 10.0.0.10:9092,10.0.0.11:9092,10.0.0.12:9092 --topic devopsdu-topic
>123
>
4：消费消息
[root@kafka-node2 ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 10.0.0.10:9092,10.0.0.11:9092,10.0.0.12:9092 --topic devopsdu-topic --from-beginning
123
