kafka 2.10安装部署
安装kafka
vi config/zookeeper.properties
vi config/server.properties
这个文件是启动kafka集群需要指定的配置文件,注意2点:
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
# The port the socket server listens on
#port=9092
listeners=PLAINTEXT://:9092
broker.id在kafka集群的每台机器上都不一样,我这里3台集群分别是0、1、2.
zookeeper.connect=master:2181,worker1:2181,worker2:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
启动kafka集群(3台都要启动)
root@master:/usr/local/kafka/kafka_2.10-0.8.2.1/bin# kafka-server-start.sh ../config/server.properties &
root@worker1:/usr/local/kafka/kafka_2.10-0.8.2.1/bin# kafka-server-start.sh ../config/server.properties &
root@worker2:/usr/local/kafka/kafka_2.10-0.8.2.1/bin# kafka-server-start.sh ../config/server.properties &
创建topic测试
./kafka-topics.sh --create --zookeeper FlinkMaster:2181,FlinkNode1:2181,FlinkNode2:2181 --replication-factor 3 --partitions 1 --topic HelloKafka
在任意服务器上运行命令为创建的Topic“HelloKafka”生产一些消息:
./kafka-console-producer.sh --broker-list FlinkMaster:9092,FlinkNode1:9092,FlinkNode2:9092 --topic HelloKafka
在任意服务器上运行命令从指定的Topic“HelloKafka”上消费(拉取)消息:
./kafka-console-consumer.sh --zookeeper FlinkMaster:2181,FlinkNode1:2181,FlinkNode2:2181 --from-beginning --topic HelloKafka