centos7环境下配置kafka_2.12-2.6.0+zookeeper-3.4.14环境
centos7环境下配置kafka_2.12-2.6.0+zookeeper-3.4.14环境
因为kafka比较稳定，考虑到服务器资源不足，单机版kafka即可满足业务需求
kafka依赖zookeeper，需要先部署zookeeper
准备工作：
添加启动kafka和zookeeper的用户
groupadd -g 1503 zookeeper
useradd -u 1503 -g zookeeper zookeeper
groupadd -g 1502 kafka
useradd -u 1502 -g kafka kafka
1.部署zookeeper
cd /usr/local/elk/
wget https://mirrors.bfsu.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar xf zookeeper-3.4.14.tar.gz
cd zookeeper-3.4.14
# 编辑zookeeper配置
# vim /usr/local/elk/zookeeper-3.4.14/conf/zoo.cfg tickTime=6000 initLimit=10 syncLimit=25 snapshot=5000 preAllocSize=1000 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/logs clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=1
# 创建相关日志和数据目录
mkdir -p /data/zookeeper/data
mkdir -p /data/zookeeper/logs
chown -R zookeeper.zookeeper /data/zookeeper
chown -R zookeeper.zookeeper /usr/local/elk/zookeeper-3.4.14
[root@eus-kafka-01:/usr/local/elk/kafka_2.12-2.1.0/config]# egrep -v '^#|^$' server.properties broker.id=249 port=9092 advertised.host.name=eus-image-design-es-kafka01 advertised.listeners=PLAINTEXT://eus-image-design-es-kafka01:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 message.max.byte=5242880 replica.fetch.max.bytes=5242880 log.dirs=/data/kafka/logs dataDir=/data/kafka/data num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 delete.topic.enable=true log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=localhost:2181 # 单zook节点 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 # 测试启动 su zookeeper bin/zkServer.sh start bin/zkServer.sh stop
# 使用systemctl管理程序，发现只能用root启动，否则报错
# vim /etc/systemd/system/zookeeper.service
[Unit]
Description=zookeeper.service
After=network.target
[Service]
Type=forking
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/elk/elasticsearch-7.4.2/jdk/bin"
User=root
Group=root
ExecStart=/usr/local/elk/zookeeper-3.4.14/bin/zkServer.sh start
ExecStop=/usr/local/elk/zookeeper-3.4.14/bin/zkServer.sh stop
PIDFile=/data/zookeeper/data/zookeeper_server.pid
[Install]
WantedBy=multi-user.target
# 用root启动是不安全的
# 修改 /usr/local/elk/zookeeper-3.4.14/bin/zkServer.sh 配置
首行添加 source /etc/profile # vim /etc/systemd/system/zookeeper.service [Unit] Description=zookeeper.service After=network.target [Service] Type=forking # Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/elk/elasticsearch-7.4.2/jdk/bin" Environment=ZOO_LOG_DIR=/data/zookeeper/logs User=zookeeper Group=zookeeper ExecStart=/usr/local/elk/zookeeper-3.4.14/bin/zkServer.sh start ExecStop=/usr/local/elk/zookeeper-3.4.14/bin/zkServer.sh stop PIDFile=/data/zookeeper/data/zookeeper_server.pid [Install] WantedBy=multi-user.target # 问题解决 systemctl daemon-reload systemctl start zookeeper
2.部署kafka
kafka下载地址：
https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.6.0/kafka_2.12-2.6.0.tgz
# 创建kafka相关的日志和存放数据目录
mkdir -p /data/kafka/{logs,data}
chown -R kafka.kafka /usr/local/elk/kafka_2.12-2.6.0
chown -R kafka.kafka /data/kafka
# 编辑配置文件
# vim /usr/local/elk/kafka_2.12-2.6.0/config/server.properties
broker.id=249 port=9092 advertised.host.name=eus-image-design-es-kafka01 advertised.listeners=PLAINTEXT://eus-image-design-es-kafka01:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 message.max.byte=5242880 replica.fetch.max.bytes=5242880 log.dirs=/data/kafka/logs dataDir=/data/kafka/data num.partitions=1 num.recovery.threads.per.data.dir=1
auto.create.topics.enable=true offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 delete.topic.enable=true log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=localhost:2181 group.initial.rebalance.delay.ms=0 zookeeper.connection.timeout.ms=600000 zookeeper.session.timeout.ms=400000
# 修改hosts(这样可以做到内网走内网，外网走外网)
# 内网的hosts配置
172.30.0.106 eus-image-design-es-kafka01
外网连接kafka的服务器配置(对应的主机和kafka advertised.listeners 对应)
kafka外网ip eus-image-design-es-kafka01
# vim /usr/local/elk/kafka_2.12-2.6.0/config/zookeeper.properties
dataDir=/data/zookeeper/data
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
#从后台启动Kafka集群测试
cd /usr/local/elk/kafka_2.12-2.6.0
su kafka
bin/kafka-server-start.sh -daemon config/server.properties
# 启动kafka报错：
[2020-08-06 20:06:37,284] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-08-06 20:06:37,288] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NumberFormatException: For input string: "2181 # lone"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68)
at java.base/java.lang.Integer.parseInt(Integer.java:658)
at java.base/java.lang.Integer.parseInt(Integer.java:776)
at org.apache.zookeeper.client.ConnectStringParser.<init>(ConnectStringParser.java:75)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1311)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:1011)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:738)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:111)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1865)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:419)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:444)
at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
解决办法：
# 去掉后面的注释即可
zookeeper.connect=localhost:2181 # 单zook节点
# 使用systemctl 管理服务
# vim /etc/systemd/system/kafka.service
[Unit] Description=Apache Kafka server (broker) After=network.target zookeeper.service [Service] Type=simple Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/elk/elasticsearch-7.4.2/jdk/bin" User=kafka Group=kafka ExecStart=/usr/local/elk/kafka_2.12-2.6.0/bin/kafka-server-start.sh /usr/local/elk/kafka_2.12-2.6.0/config/server.properties ExecStop=/usr/local/elk/kafka_2.12-2.6.0/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
# 设置开机启动
# systemctl daemon-reload
# systemctl enable kafka
# systemctl start kafka
kafka可视化工具：https://www.kafkatool.com/download.html
本地hosts解析
1.1.1.1 eus-image-design-es-kafka01
