Windows 上连接虚拟机上docker的kafka报错unknown host
Kafka Listeners - Explained
https://rmoff.net/2018/08/02/kafka-listeners-explained/
The key thing is that when you run a client, the broker you pass to it is just where it’s going to go and get the metadata about brokers in the cluster from. The actual host & IP that it will connect to for reading/writing data is based on the data that the broker passes back in that initial connection—even if it’s just a single node and the broker returned is the same as the one connected to.
You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.
Put another way, courtesy of Spencer Ruport:
LISTENERS are what interfaces Kafka binds to. ADVERTISED_LISTENERS are how clients can connect.
HOWTO: Connecting to Kafka on Docker
Here’s an example configuration:
listeners=INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092 listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://ip-172-31-18-160.us-west-2.compute.internal:19092,EXTERNAL://ec2-54-191-84-122.us-west-2.compute.amazonaws.com:9092 inter.broker.listener.name=INTERNAL
错误
[thrd:kafka1:9092/1002]: kafka1:9092/1002: Failed to resolve 'kafka1:9092': 不知道这样的主机。 (after 2253ms in state CONNECT, 1 identical error(s) suppressed)
The brokers will advertise themselve using advertised.listeners (which seems to be abstracted with KAFKA_ADVERTISED_HOST_NAME in that docker image) and the clients will consequently try to connect to these advertised hosts and ports.
You will thus need to make sure the client can resolve and reach this advertised hostnames, e.g. by adding "kafka" to /etc/hosts on the client host.
192.168.0.11 是虚拟机上的IP，Windows Client 机上连不上虚拟机里面的docker网络172.20.0.14~16，而连接里必须跟KAFKA_ADVERTISED_LISTENERS的设置地址一样，在window上hosts文件设置
192.168.0.11 kafka1
192.168.0.11 kafka2
192.168.0.11 kafka3
public override void ConfigureServices(ServiceConfigurationContext context)
{
Configure<AbpKafkaOptions>(options =>
{
//options.Connections.Default.BootstrapServers = "127.0.0.1:9092";
options.Connections.Default.BootstrapServers = "kafka1:9092,kafka2:9093,kafka3:9094";
});
Configure<AbpKafkaEventBusOptions>(options =>
{
options.GroupId = "MyGroupId2";
options.TopicName = "MyTopicName";
docker-compose.yml
version: '3.7'
networks:
br17219:
external:
name: br17219
services:
zk1:
image: zookeeper
hostname: zk1
container_name: zk1
ports:
- 2181:2181
- 8181:8080
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181
volumes:
- /usr/local/zookeeper/zk1/data:/data
- /usr/local/zookeeper/zk1/datalog:/datalog
networks:
br17219:
ipv4_address: 172.20.0.11
zk2:
image: zookeeper
hostname: zk2
container_name: zk2
ports:
- 2182:2181
- 8182:8080
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zk3:2888:3888;2181
volumes:
- /usr/local/zookeeper/zk2/data:/data
- /usr/local/zookeeper/zk2/datalog:/datalog
networks:
br17219:
ipv4_address: 172.20.0.12
zk3:
image: zookeeper
hostname: zk3
container_name: zk3
ports:
- 2183:2181
- 8183:8080
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
volumes:
- /usr/local/zookeeper/zk3/data:/data
- /usr/local/zookeeper/zk3/datalog:/datalog
networks:
br17219:
ipv4_address: 172.20.0.13
kafka1:
image: wurstmeister/kafka
depends_on:
- zk1
- zk2
- zk3
restart: always
hostname: kafka1
container_name: kafka1
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
KAFKA_LISTENERS: PLAINTEXT://kafka1:9092
volumes:
- /usr/local/kafka/kafka1/logs:/kafka
external_links:
- zk1
- zk2
- zk3
networks:
br17219:
ipv4_address: 172.20.0.14
extra_hosts:
- kafka1:172.20.0.14
kafka2:
image: wurstmeister/kafka
depends_on:
- zk1
- zk2
- zk3
restart: always
hostname: kafka2
container_name: kafka2
ports:
- 9093:9093
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka2
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093
KAFKA_LISTENERS: PLAINTEXT://kafka2:9093
volumes:
- /usr/local/kafka/kafka2/logs:/kafka
external_links:
- zk1
- zk2
- zk3
networks:
br17219:
ipv4_address: 172.20.0.15
extra_hosts:
- kafka2:172.20.0.15
kafka3:
image: wurstmeister/kafka
depends_on:
- zk1
- zk2
- zk3
restart: always
hostname: kafka3
container_name: kafka3
ports:
- 9094:9094
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka3
KAFKA_ADVERTISED_PORT: 9094
KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094
KAFKA_LISTENERS: PLAINTEXT://kafka3:9094
volumes:
- /usr/local/kafka/kafka3/logs:/kafka
external_links:
- zk1
- zk2
- zk3
networks:
br17219:
ipv4_address: 172.20.0.16
extra_hosts:
- kafka3:172.20.0.16
kafka-manager:
image: sheepkiller/kafka-manager:latest
restart: always
container_name: kafka-manager
hostname: kafka-manager
ports:
- "9010:9000"
links:
- kafka1
- kafka2
- kafka3
external_links:
- zk1
- zk2
- zk3
environment:
ZK_HOSTS: zk1:2181,zk2:2181,zk3:2181
TZ: CST-8
networks:
br17219:
ipv4_address: 172.20.0.20
efak:
image: ydockerp/efak:2.0.8
restart: always
container_name: efak-manager
hostname: efak
ports:
- "8048:8048"
depends_on:
- kafka1
- kafka2
- kafka3
environment:
ZK_HOSTS: zk1:2181,zk2:2181,zk3:2181
networks:
br17219:
ipv4_address: 172.20.0.21
