docker部署Kafka集群(Kafka新版本,不使用zookeeper)

参考-->官方文档

主机ip	服务
192.168.86.59	Kafka
192.168.86.20	Kafka
192.168.86.43	Kafka

创建

在主机为192.168.86.59上的docker-compose.yaml

version: "3"
services:
   kafka:
     image: 'bitnami/kafka:latest'
     user: root
     ports:
       - '9092:9092'
     environment:
       - KAFKA_ENABLE_KRAFT=yes
       - KAFKA_CFG_PROCESS_ROLES=broker,controller
       - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
       - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
       - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
       - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.86.59:9092
       - KAFKA_BROKER_ID=1
       - KAFKA_KRAFT_CLUSTER_ID=LelM2dIFQkiUFvXCEcqRWA
       - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@192.168.86.59:9093,2@192.168.86.20:9093,3@192.168.86.43:9093
       - ALLOW_PLAINTEXT_LISTENER=yes
     volumes:
       - /data/deploy/kafkaCluster/kraft:/bitnami/kafka:rw
     network_mode: host

启动: docker-compose up -d

在192.168.86.20上的docker-compose.yaml

version: "3"
services:
   kafka:
     image: 'bitnami/kafka:latest'
     user: root
     ports:
       - '9092:9092'
     environment:
       - KAFKA_ENABLE_KRAFT=yes
       - KAFKA_CFG_PROCESS_ROLES=broker,controller
       - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
       - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
       - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
       - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.86.20:9092
       - KAFKA_BROKER_ID=2
       - KAFKA_KRAFT_CLUSTER_ID=LelM2dIFQkiUFvXCEcqRWA
       - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@192.168.86.59:9093,2@192.168.86.20:9093,3@192.168.86.43:9093
       - ALLOW_PLAINTEXT_LISTENER=yes
     volumes:
       - /data/deploy/kafkaCluster/kraft:/bitnami/kafka:rw
     network_mode: host

启动:docker-compose up -d

在192.168.86.43上的docker-compose.yaml

version: "3"
services:
   kafka:
     image: 'bitnami/kafka:latest'
     user: root
     ports:
       - '9092:9092'
     environment:
       - KAFKA_ENABLE_KRAFT=yes
       - KAFKA_CFG_PROCESS_ROLES=broker,controller
       - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
       - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
       - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
       - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.86.43:9092
       - KAFKA_BROKER_ID=3
       - KAFKA_KRAFT_CLUSTER_ID=LelM2dIFQkiUFvXCEcqRWA
       - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@192.168.86.59:9093,2@192.168.86.20:9093,3@192.168.86.43:9093
       - ALLOW_PLAINTEXT_LISTENER=yes
     volumes:
       - /data/deploy/kafkaCluster/kraft:/bitnami/kafka:rw
     network_mode: host

启动: docker-compose up -d

测试

1.在Broker 1上创建一个副本为3、分区为5的topic用于测试
因Kafka的topic所有分区会分散在不同Broker上，所以该topic的5个分区会被分散到3个Broker上，其中有两个Broker得到两个分区，另一个Broker只有1个分区。该结论在下面将会得到验证
cd bin

kafka-topics.sh --create --topic foo --partitions 5 --replication-factor 3 --bootstrap-server 192.168.86.59:9092,192.168.86.20:9092,192.168.86.43:9092

查看topic列表
kafka-topics.sh --list --bootstrap-server 192.168.86.59:9092,192.168.86.20:9092,192.168.86.43:9092

查看topic详细信息
kafka-topics.sh --describe --topic foo --bootstrap-server 192.168.86.59:9092,192.168.86.20:9092,192.168.86.43:9092

开启消费者
kafka-console-consumer.sh --bootstrap-server 192.168.86.59:9092,192.168.86.20:9092,192.168.86.43:9092 --topic foo
我在192.168.86.59和192.168.20上开启消费者,在192.168.86.43上开启生产者

开启生产者
kafka-console-producer.sh --broker-list 192.168.86.59:9092,192.168.86.20:9092,192.168.86.43:9092 --topic foo
在43上随便输入一个值

在59和20上查看

删除topic

只有生产者和消费都退出了才能删除
kafka-topics.sh --delete --topic foo --bootstrap-server 192.168.86.59:9092,192.168.86.20:9092,192.168.86.43:9092