docker-compose 搭建 kafka集群
搭建集群
docker-compose up -d --scale kafka=2 本机启动一个有两个节点的 Kafka 集群
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092" # kafka 把9092端口随机映射到主机的端口
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.199.163 #本机ip
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: test:1:1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
输出结果
chen@macos ~/docker/kafka01 sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4ede7dc26df7 wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 22 minutes ago Up 22 minutes 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp kafka01_zookeeper_1
2f7c0a71af03 wurstmeister/kafka "start-kafka.sh" 22 minutes ago Up 22 minutes 0.0.0.0:32772->9092/tcp kafka01_kafka_2
e0b72d9466e0 wurstmeister/kafka "start-kafka.sh" 22 minutes ago Up 22 minutes 0.0.0.0:32771->9092/tcp kafka01_kafka_1
注意:docker-compose up -d --scale kafka=2 # 可以进行扩展或缩容
基础操作
列出主题
容器内部查询主题
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181 #可以通过容器的名字或者IP访问
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
容器外部或内部部查询主题
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper 192.168.199.163:2181 #这里因为zookeeper映射到了主题的2181端口
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
创建主题
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic chat --partitions 3 --zookeeper 192.168.199.163:2181 --replication-factor 2
Created topic chat.
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181
Topic:chat PartitionCount:3 ReplicationFactor:2 Configs:
Topic: chat Partition: 0 Leader: 1002 Replicas: 1002,1001 Isr: 1002,1001
Topic: chat Partition: 1 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002
Topic: chat Partition: 2 Leader: 1002 Replicas: 1002,1001 Isr: 1002,1001
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
消息消费
生产者-容器里面
bash-4.4# ./kafka-console-producer.sh --broker-list localhost:9092 --topic chat
>da
>
消费者-容器里面
/opt/kafka_2.12-2.2.0/bin # ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic chat --from-beginning
da
注意:上面要在宿主机上只要将localhost:9092换为kafka暴露在主机上端口即可