kafka2.8集群搭建及监控配置

背景

业务需要搭建kafka集群及监控，kafka依赖于zookeeper集群，所以也一并记录下来。

ip	操作系统	主机名	用途
192.168.0.19	CentOS7	zk1	zookeeper集群
192.168.0.36	CentOS7	zk2	zookeeper集群
192.168.0.18	CentOS7	zk3	zookeeper集群
192.168.0.137	CentOS7	kafka01	kafka集群
192.168.0.210	CentOS7	kafka02	kafka集群
192.168.0.132	CentOS7	kafka03	kafka集群

zookeeper集群搭建

搭建

3台都要操作

下载安装包，解压

wget https://mirrors.cloud.tencent.com/apache/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
tar zxvf apache-zookeeper-3.7.0-bin.tar.gz
mv apache-zookeeper-3.7.0-bin zookeeper
rm -f apache-zookeeper-3.7.0-bin.tar.gz 

创建集群配置文件

cat <<EOF> /data/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data/
clientPort=2181
server.0=192.168.0.19:2888:3888
server.1=192.168.0.36:2888:3888
server.2=192.168.0.18:2888:3888
EOF

创建数据目录

mkdir -p /data/zookeeper/data/

3台分别操作

zk1

echo 0 > /data/zookeeper/data/myid

zk2

echo 1 > /data/zookeeper/data/myid

zk3

echo 2 > /data/zookeeper/data/myid

3台都执行

cd /data/zookeeper/bin/ && ./zkServer.sh start
cd /data/zookeeper/bin/ && ./zkServer.sh status

会输出如下结果：

# ./zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

# ./zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

# ./zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

搭建完成

kafka集群搭建

搭建

以下操作3台都要做

下载安装包，解压

cd /data
wget https://mirrors.cloud.tencent.com/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz
tar xvf kafka_2.13-2.8.0.tgz
mv kafka* kafka

配置环境变量

cat <<EOF> /etc/profile.d/kafka.sh
export KAFKA_HOME=/data/kafka
export PATH=$PATH:$KAFKA_HOME/bin
EOF

source /etc/profile.d/kafka.sh

重启脚本

cat <<EOF> /data/kafka/restart.sh
#!/bin/bash

kafka-server-stop.sh
nohup kafka-server-start.sh config/server.properties >> /data/kafka/nohup.out 2>&1 &
EOF

chmod +x /data/kafka/restart.sh

用于监控的配置，修改 bin/kafka-server-start.sh，增加 JMX_PORT，可以获取更多指标

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT="9099"
fi

关键配置如下：

name	含义	举例
broker.id	一个Kafka节点就是一个Broker.id，要保证唯一性	broker.id=0
listeners	kafka只面向内网时用到listeners，内外网需要作区分时才需要用到advertised.listeners	listeners=PLAINTEXT://192.168.0.137:9092
zookeeper.connect	配置zk集群信息	zookeeper.connect=192.168.0.19:2181

kafka01 配置文件/data/kafka/config/server.properties

broker.id=0
listeners=PLAINTEXT://192.168.0.137:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.0.19:2181,192.168.0.36:2181,192.168.0.18:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

kafka02 配置文件/data/kafka/config/server.properties

broker.id=1
listeners=PLAINTEXT://192.168.0.210:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.0.19:2181,192.168.0.36:2181,192.168.0.18:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

kafka03 配置文件/data/kafka/config/server.properties

broker.id=2
listeners=PLAINTEXT://192.168.0.132:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.0.19:2181,192.168.0.36:2181,192.168.0.18:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

3台都启动

/data/kafka/restart.sh

监控

官方主页：https://github.com/smartloli/kafka-eagle

官方部署文档：https://www.kafka-eagle.org/articles/docs/installation/linux-macos.html

安装包下载地址：

https://github.com/smartloli/kafka-eagle/archive/refs/tags/v2.0.6.tar.gz

安装配置

mkdir -p /opt/kafka-eagele
tar zxvf kafka-eagle-bin-2.0.6.tar.gz 
cd kafka-eagle-bin-2.0.6/
tar zxvf kafka-eagle-web-2.0.6-bin.tar.gz
mv kafka-eagle-web-2.0.6/* /opt/kafka-eagele/

配置文件 conf/system-config.properties

# 填写zk地址，会自动获取到kafka节点
kafka.eagle.zk.cluster.alias=cluster1
cluster1.zk.list=tdn1:2181,tdn2:2181,tdn3:2181

# 默认使用sqlite，容易死锁，需修改为MySQL
# Default use sqlite to store data
#kafka.eagle.driver=org.sqlite.JDBC
# It is important to note that the '/hadoop/kafka-eagle/db' path must be exist.
#kafka.eagle.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
#kafka.eagle.username=root
#kafka.eagle.password=smartloli

# MySQL创建ke数据库即可，无需导 SQL
kafka.eagle.driver=com.mysql.jdbc.Driver
kafka.eagle.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=smartloli

运行

cd bin
chmod +x ke.sh 
./ke.sh start

浏览器访问 http://localhost:8048

admin/123456