CentOS部署Kafka中间件

CentOS部署Kafka中间件

 

1.环境及版本说明：
　　系统版本：CentOS Linux release 7.6.1810 (Core)
　　Kafka版本：kafka_2.12-2.2.0
　　JDK版本：1.8.0_212
2.安装

下载

# wget下载安装包
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz

#无法下载尝试更换其他下载地址
wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz

#通过网盘分享的文件：kafka_2.12-2.2.0.tgz
#链接: https://pan.baidu.com/s/1V99Kqsd27OWPBHv0K8ACEg?pwd=27p2 提取码: 27p2

创建安装目录

mkdir /usr/local/kafka

解压到安装目录

tar -zxvf kafka_2.12-2.2.0.tgz -C /usr/local/kafka/

3.编辑Kafka、Zookeeper配置文件

#编辑Kafka配置文件
vi /usr/local/kafka/kafka_2.12-2.2.0/config/server.properties 

log.dirs=/usr/local/kafka/kafka_2.12-2.2.0/kafka-logs  //日志文件存放目录
#远程连接
#去掉31行的注释，listeners=PLAINTEXT://:9092
#去掉36行的注释，把advertised.listeners值改为PLAINTEXT://host.name:9092（host.name是你的IP地址）

#编辑Zookeeper配置文件
vi /usr/local/kafka/kafka_2.12-2.2.0/config/zookeeper.properties 
#修改数据存放目录位置
dataDir=/usr/local/kafka/kafka_2.12-2.2.0/zookeeper_data

4.启动命令

启动Zookeeper

nohup /usr/local/kafka/kafka_2.12-2.2.0/bin/zookeeper-server-start.sh /usr/local/kafka/kafka_2.12-2.2.0/config/zookeeper.properties  >zookeeper.log 2>&1 &

启动Kafka

nohup /usr/local/kafka/kafka_2.12-2.2.0/bin/kafka-server-start.sh  /usr/local/kafka/kafka_2.12-2.2.0/config/server.properties >kafka.log 2>&1 &

5.测试

使用kafka，创建一个主题（topic）创建一个名为“test”的Topic，只有一个分区和一个备份

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看已创建的topic信息

bin/kafka-topics.sh --list --zookeeper localhost:2181

发送消息

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

消费消息

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

6.一键安装脚本

#!/bin/bash

wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz
mkdir /usr/local/kafka
tar -zxvf kafka_2.12-2.2.0.tgz -C /usr/local/kafka/
network=`ip a | grep '2: ' | awk {'print $2'} | sed "s/://g"`
ip=`ifconfig $network |grep -w 'inet' | awk '{print $2}'`
if [ 0 -eq $? ] ; then
        sed -i "/^#listeners=PLAINTEXT/clisteners=PLAINTEXT://$ip:9092" "/usr/local/kafka/kafka_2.12-2.2.0/config/server.properties"
        sed -i "/^log.dir/clog.dirs=/usr/local/kafka/kafka_2.12-2.2.0/kafka-logs" "/usr/local/kafka/kafka_2.12-2.2.0/config/server.properties"
        sed -i "/^#advertised.listeners/cadvertised.listeners=PLAINTEXT://$ip:9092" "/usr/local/kafka/kafka_2.12-2.2.0/config/server.properties"
else
        echo "No geting IP! Kafka config need reconfigure!"
fi
nohup /usr/local/kafka/kafka_2.12-2.2.0/bin/zookeeper-server-start.sh /usr/local/kafka/kafka_2.12-2.2.0/config/zookeeper.properties  >zookeeper.log 2>&1 &
nohup /usr/local/kafka/kafka_2.12-2.2.0/bin/kafka-server-start.sh  /usr/local/kafka/kafka_2.12-2.2.0/config/server.properties >kafka.log 2>&1 &

7.参考借鉴

https://cloud.tencent.com/developer/article/2445341?from=15425

https://cloud.tencent.com/developer/article/1986967