手动部署Kraft模式kafka集群
基本信息
IP地址 |
Hostname |
Release |
Kafka-Version |
172.29.145.157 |
iamdemo1 |
Centos7.9 |
kafka_2.12-3.5.1 |
172.29.145.182 |
iamdemo2 |
Centos7.9 |
kafka_2.12-3.5.1 |
172.29.145.183 |
iamdemo3 |
Centos7.9 |
kafka_2.12-3.5.1 |
下载安装包
kafka安装包官网下载
下载完成后上传到服务器/opt目录下解压
| tar -xf kafka_2.12-3.5.1.tgz |
| 将文件夹重命名为kafka |
| mv kafka_2.12-3.5.1 kafka |
生成集群随机uuid
| [root@iamdemo1 ~ ]cd /opt/kafka/bin |
| [root@iamdemo1 bin]# sh kafka-storage.sh random-uuid |
| COY-4PxTT82yfvGKqAepwQ |
配置kafka集群的kraft模式参数
| 参数位置:/opt/kafka/config/kraft/server.properties |
| # |
| process.roles=broker,controller |
| # |
| node.id=1 |
| # |
| controller.quorum.voters=1@172.29.145.157:9093,2@172.29.145.182:9093,3@172.29.145.183:9093 |
| # |
| listeners=PLAINTEXT://:9092,CONTROLLER://:9093 #,EXTERNAL://172.29.145.157:9094 |
| # |
| inter.broker.listener.name=PLAINTEXT |
| # |
| #advertised.listeners=EXTERNAL://172.29.145.157:9094,PLAINTEXT://:9092 |
| advertised.listeners=PLAINTEXT://172.29.145.157:9092 |
| controller.listener.names=CONTROLLER |
| listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,EXTERNAL:PLAINTEXT |
| num.network.threads=3 |
| num.io.threads=8 |
| socket.send.buffer.bytes=102400 |
| socket.receive.buffer.bytes=102400 |
| socket.request.max.bytes=104857600 |
| # |
| log.dirs=/opt/kafka/kraft-combined-logs |
| # |
| num.partitions=6 |
| # |
| num.recovery.threads.per.data.dir=2 |
| # |
| offsets.topic.replication.factor=3 |
| # |
| transaction.state.log.replication.factor=3 |
| transaction.state.log.min.isr=3 |
| auto.create.topics.enable=false |
| default.replication.factor=3 |
| # |
| log.retention.hours=168 |
| # |
| log.segment.bytes=1073741824 |
| # |
| log.retention.check.interval.ms=300000 |
使用集群uuid格式化kafka数据存储目录
| [root@iamdemo1 bin]# sh kafka-storage.sh format -t COY-4PxTT82yfvGKqAepwQ -c ../config/kraft/server.properties |
| Formatting /opt/kafka/kraft-combined-logs with metadata.version 3.5-IV2. |
修改启动脚本,暴露JMX端口,配置JVM内存大小
| # |
| if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then |
| export KAFKA_HEAP_OPTS="-Xmx4G -Xms4G" |
| export JMX_PORT="9999" |
| fi |
修改kafka日志配置参数,避免一小时生成一个日志文件
| 文件位置:/opt/kafka/config/log4j.properties |
| # |
| log4j.appender.kafkaAppender.MaxFileSize=100MB |
| log4j.appender.kafkaAppender.MaxBackupIndex=5 |
| # |
| log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd |
使用脚本启动kafka集群
| [root@iamdemo1 bin]# sh kafka-server-start.sh -daemon ../config/kraft/server.properties |
查看集群状态
| 通过jps即可看到kafka进程是否在运行,也可通过ps aux查看 |
| 查看topic副本分布情况可以通过 |
| ./kafka-topics.sh --describe --bootstrap-server 172.29.145.157:9092,172.29.145.182:9092,172.29.145.183:9092 |
| |
| Topic: test-topic1 TopicId: U3-YpQ4QQqWM7jQ-8uAxjA PartitionCount: 6 ReplicationFactor: 3 Configs: segment.bytes=1073741824 |
| Topic: test-topic1 Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 |
| Topic: test-topic1 Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 |
| Topic: test-topic1 Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 |
| Topic: test-topic1 Partition: 3 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 |
| Topic: test-topic1 Partition: 4 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 |
| Topic: test-topic1 Partition: 5 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 |
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· 清华大学推出第四讲使用 DeepSeek + DeepResearch 让科研像聊天一样简单!
· 实操Deepseek接入个人知识库
· CSnakes vs Python.NET:高效嵌入与灵活互通的跨语言方案对比
· Plotly.NET 一个为 .NET 打造的强大开源交互式图表库