Centos7的kafka集群搭建

CentOS 7搭建kafka集群

原创莫问记录栈 2022-11-16 19:49 发表于陕西

收录于合集

#centos6个

#kafka1个

#linux9个

#kafka集群1个

/**

*@系统：CentOS Linux release 7.9.2009 (Core)

*@说明：jdk-11.0.12 | apache-zookeeper-3.7.1 | kafka_2.12-3.3.1

* 注意：因为有大部分操作是相同的，所以使用的是工具的批操作

一、准备工作

1. 下载 jdk11

https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html#license-lightbox

2. 下载 zookeeper

https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz

3. 下载 kafka

https://downloads.apache.org/kafka/3.3.1/kafka_2.12-3.3.1.tgz

4. 修改主机名

# hostnamectl set-hostname kafka01# hostnamectl set-hostname kafka02# hostnamectl set-hostname kafka03

5. 修改 hosts

# vim /etc/hosts==== content ====192.168.137.129    kafka01192.168.137.130    kafka02192.168.137.131    kafka03==== / ====

6. 开放防火墙端口

# 生产环境注意修改zookeeper和kafka的端口# firewall-cmd --zone=public --add-port=2881/tcp --permanent# firewall-cmd --zone=public --add-port=3881/tcp --permanent# firewall-cmd --zone=public --add-port=9092/tcp --permanent# firewall-cmd --reload

二、安装 jdk11

1. 安装

# rpm -ivh jdk-11.0.12_linux-x64_bin.rpm

2. 添加环境变量

# ls /usr/java# vim /etc/profile==== content ====export JAVA_HOME=/usr/java/jdk-11.0.12export PATH=$PATH:$JAVA_HOME/bin==== / ====# . /etc/profile ##source /etc/profile

3. 验证

# java -versionjava version "11.0.12" 2021-07-20 LTSJava(TM) SE Runtime Environment 18.9 (build 11.0.12+8-LTS-237)Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.12+8-LTS-237, mixed mode)# cd $JAVA_HOME# pwd/usr/java/jdk-11.0.12

三、安装 zookeeper

1. 安装

# tar -zxf apache-zookeeper-3.7.1-bin.tar.gz -C /usr/local# cd /usr/local/# mv apache-zookeeper-3.7.1-bin zookeeper

2. 修改配置文件

# cd /usr/local/zookeeper/conf/# cp zoo_sample.cfg zoo.cfg# mkdir -p /var/zookeeper/data# 在对应的三台机子上分别执行# kafka01# vim zoo.cfg==== content ====dataDir=/var/zookeeper/dataserver.1=0.0.0.0:2881:3881server.2=192.168.137.130:2881:3881server.3=192.168.137.131:2881:3881==== / ====# echo 1 > /var/zookeeper/data/myid# cat /var/zookeeper/data/myid1# kafka02# vim zoo.cfg==== content ====dataDir=/var/zookeeper/dataserver.1=192.168.137.129:2881:3881server.2=0.0.0.0:2881:3881server.3=192.168.137.131:2881:3881==== / ====# echo 2 > /var/zookeeper/data/myid# cat /var/zookeeper/data/myid2# kafka03# vim zoo.cfg==== content ====dataDir=/var/zookeeper/dataserver.1=192.168.137.129:2881:3881server.2=192.168.137.130:2881:3881server.3=0.0.0.0:2881:3881==== / ====# echo 3 > /var/zookeeper/data/myid# cat /var/zookeeper/data/myid3

3. 添加系统服务

# cd /etc/systemd/system# vim zookeeper.service==== content ====[Unit]Description=Zookeeper Service
[Service]Type=simpleWorkingDirectory=/usr/local/zookeeper/PIDFile=/var/zookeeper/data/zookeeper_server.pidSyslogIdentifier=zookeeperExecStart=/usr/local/zookeeper/bin/zkServer.sh startExecStop=/usr/local/zookeeper/bin/zkServer.sh stopRestart=alwaysTimeoutSec=20SuccessExitStatus=130 143
[Install]WantedBy=multi-user.target==== / ====# systemctl daemon-reload# systemctl enable zookeeper.serviceCreated symlink from /etc/systemd/system/multi-user.target.wants/zookeeper.service to /etc/systemd/system/zookeeper.service.

四、安装 kafka

1. 安装

# tar -zxf kafka_2.13-3.3.1.tgz -C /usr/local/# mv kafka_2.13-3.3.1 kafka

2. 修改配置文件

# cd /usr/local/kafka/config# vim server.properties==== content ====# kafka01broker.id=0# listeners=PLAINTEXT://:9092listeners=PLAINTEXT://192.168.137.129:9092# advertised.listeners=PLAINTEXT://your.host.name:9092advertised.listeners=PLAINTEXT://192.168.137.129:9092# log.dirs=/tmp/kafka-logslog.dirs=/var/kafka/logs# zookeeper.connect=localhost:2181zookeeper.connect=192.168.137.129:2181,192.168.137.130:2181,192.168.137.131:2181/mykafka----------# kafka02broker.id=1# listeners=PLAINTEXT://:9092listeners=PLAINTEXT://192.168.137.130:9092# advertised.listeners=PLAINTEXT://your.host.name:9092advertised.listeners=PLAINTEXT://192.168.137.130:9092# log.dirs=/tmp/kafka-logslog.dirs=/var/kafka/logs# zookeeper.connect=localhost:2181zookeeper.connect=192.168.137.129:2181,192.168.137.130:2181,192.168.137.131:2181/mykafka----------# kafka03broker.id=2# listeners=PLAINTEXT://:9092listeners=PLAINTEXT://192.168.137.131:9092# advertised.listeners=PLAINTEXT://your.host.name:9092advertised.listeners=PLAINTEXT://192.168.137.131:9092# log.dirs=/tmp/kafka-logslog.dirs=/var/kafka/logs# zookeeper.connect=localhost:2181zookeeper.connect=192.168.137.129:2181,192.168.137.130:2181,192.168.137.131:2181/mykafka==== / ====

3. 添加系统服务

# cd /etc/systemd/system# vim kafka.service==== content ====[Unit]Requires=zookeeper.serviceAfter=zookeeper.service
[Service]Type=simpleExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.propertiesExecStop=/usr/local/kafka/bin/kafka-server-stop.shRestart=AlwaysTimeoutSec=20
[Install]WantedBy=multi-user.target==== / ====
# systemctl daemon-reload# systemctl enable kafka.serviceCreated symlink from /etc/systemd/system/multi-user.target.wants/kafka.service to /etc/systemd/system/kafka.service.# init 6

4. 验证

# kafka01# jps8977 Jps716 QuorumPeerMain8623 Kafka
# kafka02# jps8251 Kafka718 QuorumPeerMain8607 Jps

# kafka03# jps8994 Kafka9315 Jps712 QuorumPeerMain
# systemctl status zookeeper.service● zookeeper.service - Zookeeper Service   Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled)   Active: active (running) # systemctl status kafka.service● kafka.service - Kafka service   Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)   Active: active (running)