kafka集群搭建

kafka集群搭建

1、环境准备

IP	HostName	应用	安装路径
192.168.0.81	node81	zookeeper、kafka	/opt
192.168.0.82	node82	zookeeper、kafka	/opt
192.168.0.83	node83	zookeeper、kafka	/opt

软件包下载

软件	版本	安装节点	目录	下载地址
jdk	jdk-17_linux-x64_bin.rpm	node81,node82,node83	/usr/java/	https://www.oracle.com/java/technologies/downloads/
zookeeper	apache-zookeeper-3.5.6-bin.tar.gz	node81,node82,node83	/opt/zookeeper	https://archive.apache.org/dist/zookeeper/zookeeper-3.5.6/
kafka	kafka_2.13-3.1.0.tgz	node81,node82,node83	/opt/kafka	https://kafka.apache.org/downloads
eagle	kafka-eagle-bin-2.1.0.tar.gz	node81	/opt/kafka-eagle	http://download.kafka-eagle.org/

创建普通用户

useradd hzyyg
passwd hzyyg

2、安装

2.1、安装jdk

上传安装包到/usr/local/src目录下

cd /usr/local/src
rpm -ivh jdk-17_linux-x64_bin.rpm

添加java环境变量

#vim /etc/profile
export JAVA_HOME=/usr/java/jdk-17.0.2
export CLASSPATH=.:${JAVA_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
#source /etc/profile

测试java

#java -version
java version "17.0.2" 2022-01-18 LTS
Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)

2.2 安装zookeeper

修改内存配置

#vim /etc/security/limits.conf
hzyyg  soft nproc 16384
hzyyg  hard nproc 16384
hzyyg  soft nofile 65535
hzyyg  hard nofile 65536

解压

cd /usr/local/src
mkdir /opt/zookeeper
tar -zxvf apache-zookeeper-3.5.6-bin.tar.gz -C /opt/zookeeper/

创建软连接，后期方便更新版本

ln -s /opt/zookeeper/apache-zookeeper-3.5.6-bin /opt/zookeeper/zookeeper

添加环境变量

echo -e 'export ZOOKEEPER_HOME=/opt/zookeeper/zookeeper\nexport PATH=$ZOOKEEPER_HOME/bin/$PATH' >> /etc/profile

配置zk配置文件

cp /opt/zookeeper/zookeeper/conf/zoo_sample.cfg  /opt/zookeeper/zookeeper/conf/zoo.cfg 
echo 'server.1=192.168.0.81:2888:3888
server.2=192.168.0.82:2888:3888
server.3=192.168.0.83:2888:3888' >> /opt/zookeeper/zookeeper/conf/zoo_sample.cfg 
mkdir /opt/zookeeper/data

分别设置myid,node82和83分别设置为2 ，3

#node81
echo 1 > /opt/zookeeper/data/myid

启动zookeeper

/opt/zookeeper/zookeeper/bin/zkServer.sh start

3、安装kafka

解压

cd /usr/local/src
mkdir /opt/kafka
tar -zxvf kafka_2.13-3.1.0.tgz
tar -xzvf kafka-eagle-bin-2.1.0/efak-web-2.1.0-bin.tar.gz -C /opt/kafka-eagle/

创建软连接

ln -s /opt/kafka/kafka_2.13-3.1.0 /opt/kafka/kafka

环境变量

echo -e 'KAFKA_HOME=/opt/kafka/apache-kafka \nexport PATH=$KAFKA_HOME/bin:$PATH' >> /etc/profile

修改server.conf

broker.id=1
listeners=PLAINTEXT://192.168.0.81:9092
log.dirs=/opt/kafka/kafka-logs
zookeeper.connect=192.168.0.81:2181,192.168.0.82:2181,192.168.0.83:2181

broker.id=2
listeners=PLAINTEXT://192.168.0.82:9092
log.dirs=/opt/kafka/kafka-logs
zookeeper.connect=192.168.0.81:2181,192.168.0.82:2181,192.168.0.83:2181

broker.id=3
listeners=PLAINTEXT://192.168.0.83:9092
log.dirs=/opt/kafka/kafka-logs
zookeeper.connect=192.168.0.81:2181,192.168.0.82:2181,192.168.0.83:2181

启动kafka

su hzyyg
sudo bin/kafka-server-start.sh -daemon config/server.properties

kafka测试

创建topic

bin/kafka-topics.sh  --create --bootstrap-server 192.168.0.81:9092 --replication-factor 1 --partitions 1 --topic test1
bin/kafka-topics.sh --list --bootstrap-server 192.168.0.81:9092

发送消息/生产消息

bin/kafka-console-producer.sh --broker-list 192.168.0.81:9092 --topic test
>hello world

消费消息

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.81:9092 --topic test --from-beginning
hello world

4、安装kafka-eagle

cd /usr/local/src
mkdir /opt/kafka-eagle
tar -zxvf kafka-eagle-bin-2.1.0.tar.gz -C /opt/kafka-eagle/

创建软连接

ln -s /opt/kafka-eagle/efak-web-2.1.0 /opt/kafka-eagle/efak-web

添加环境变量

echo 'export KE_HOME=/opt/kafka-eagle/kafka-eagle' >> /etc/profile

修改配置文件

#vim /opt/kafka-eagle/efak-web/conf/system-config.properties

efak.zk.cluster.alias=cluster1
cluster1.zk.list=192.168.0.81:2181,192.168.0.82:2181,192.168.0.83:2181

######################################
# kafka mysql jdbc driver address
######################################
efak.driver=com.mysql.cj.jdbc.Driver
efak.url=jdbc:mysql://192.168.31.247:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
efak.username=root
efak.password=123456

修改kafka JMX配置

sed -i '/export KAFKA_HEAP_OPTS/a\    export JMX_PORT="7788"' /opt/kafka/kafka/bin/kafka-server-start.sh
cat  /opt/kafka/kafka/bin/kafka-server-start.sh | grep -C 2 'export KAFKA_HEAP_OPTS'

启动kafka-eagle

/opt/kafka-eagle/efak-web/bin/ke.sh start

用户名密码：admin / 123456