Kafka架构

producer: 生产者
consumer: 消费者
broker: 类似容器,存放数据
topic: 类似标签

Kafka部署

单节点单Broker部署和使用

1.创建文件夹

mkdir -p /usr/local/kafka/data  /usr/local/kafka/logs
cd /usr/local/kafka

2.下载Flume http://kafka.apache.org/downloads

wget http://apache.mirror.iphh.net/kafka/2.5.0/kafka_2.13-2.5.0.tgz

3.解压

tar -zvxf kafka_2.13-2.5.0.tgz

4.修改配置文件

vim /etc/profile
export KAFKA_HOME=/usr/local/kafka/kafka_2.13-2.5.0
export PATH=$KAFKA_HOME/bin:$PATH

source /etc/profile

5.修改配置

cd /usr/local/kafka/kafka_2.13-2.5.0/config
vim server.properties
broker.id=0   # broker id 这个不能重复
listeners=PLAINTEXT://:9092   # 取消注释  默认端口是9092
host.name=0.0.0.0   # ip
log.dirs=/usr/local/kafka/logs  # log文件放置路径
zookeeper.connect=localhost:2181   # zookeeper连接信息

6.启动kafka

kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties
-daemon: 后台启动
config/server.properties: server.properties文件路径
kafka-server-stop.sh  # 关闭kafka

jps  查看进程  jps -m: 查看比较详细的信息
kafka

启动报错: kafka.common.InconsistentClusterIdException: The Cluster ID BmhgroWiQEWltKkCY_4u4Q doesn't match stored clusterId Some(3BVX0S4nTjSvbCvqUU3WGQ) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
    在Kafka config目录中，打开kafka config属性文件，让server.properties查找具有参数log.dirs =的日志路径目录，然后转到日志路径目录并在其中找到文件meta.properties。打开文件meta.properties并更新cluster.id =或从日志路径目录中删除此文件或所有日志文件，然后重新启动kafka。（这个方法并没有解决我的问题）

kafka-server-stop.sh $KAFKA_HOME/config/server.properties  # 关闭kafka

cd /usr/local/kafka/kafka_2.13-2.5.0/logs
cat server.log | tail -n 200  # 查看kafka启动时产生日志

7.创建一个topic

kafka-topics.sh --create --bootstrap-server 192.168.104.94:9092 --replication-factor 1 --partitions 1 --topic hello_topic
--create: 创建
--replication-factor: 副本数量
--partitions: 分区数量
--bootstrap-server: 制定消费者

kafka-topics.sh --delete --bootstrap-server 192.168.104.94:9092 --topic hello_topic  # 删除topic

8.查看所有topic

kafka-topics.sh --list --bootstrap-server 192.168.104.94:9092

9.发送消息

kafka-console-producer.sh --bootstrap-server 192.168.104.94:9092 --topic hello_topic  # 生产消息
kafka-console-consumer.sh --bootstrap-server 192.168.104.94:9092 --topic hello_topic --from-beginning   # 消费消息  # --from-beginning: 从头开始消费  每次启动都要把一起的消息在消费一遍

10.查看topic信息

kafka-topics.sh --describe --bootstrap-server 192.168.104.94:9092 --topic hello_topic  # --topic hello_topic: 不带这个就看所有的topic

单节点多Broker部署和使用

1.创建文件夹

mkdir -p /usr/local/kafka/data  /usr/local/kafka/logs /usr/local/kafka/logs1 /usr/local/kafka/logs2 /usr/local/kafka/logs3
cd /usr/local/kafka

2.下载Flume http://kafka.apache.org/downloads

wget http://apache.mirror.iphh.net/kafka/2.5.0/kafka_2.13-2.5.0.tgz

3.解压

tar -zvxf kafka_2.13-2.5.0.tgz

4.修改配置文件

vim /etc/profile
export KAFKA_HOME=/usr/local/kafka/kafka_2.13-2.5.0
export PATH=$KAFKA_HOME/bin:$PATH

source /etc/profile

5.修改配置

cd /usr/local/kafka/kafka_2.13-2.5.0/config
cp server.properties server1.properties
cp server.properties server2.properties
cp server.properties server3.properties
vim server.properties
broker.id=0   # broker id 这个不能重复  1, 2, 3 往上设置
listeners=PLAINTEXT://:9092   # 取消注释  默认端口是9092   修改三个配置文件 端口不能重复9092 9093 9094 9095
host.name=0.0.0.0   # ip
log.dirs=/usr/local/kafka/logs  # log文件放置路径   修改三个配置文件 路径不能重复 上次建了三个文件路径分布配置这三个文件路径
zookeeper.connect=localhost:2181   # zookeeper连接信息

6.启动kafka

kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server1.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server2.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server3.properties &
-daemon: 后台启动
config/server.properties: server.properties文件路径
kafka-server-stop.sh  # 关闭kafka

jps  查看进程  jps -m: 查看比较详细的信息
kafka

启动报错: kafka.common.InconsistentClusterIdException: The Cluster ID BmhgroWiQEWltKkCY_4u4Q doesn't match stored clusterId Some(3BVX0S4nTjSvbCvqUU3WGQ) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
    在Kafka config目录中，打开kafka config属性文件，让server.properties查找具有参数log.dirs =的日志路径目录，然后转到日志路径目录并在其中找到文件meta.properties。打开文件meta.properties并更新cluster.id =或从日志路径目录中删除此文件或所有日志文件，然后重新启动kafka。（这个方法并没有解决我的问题）

cd /usr/local/kafka/kafka_2.13-2.5.0/logs
cat server.log | tail -n 200  # 查看kafka启动时产生日志

多节点多Broker部署和使用

0. 服务器地址  先安装zookeeper集群 https://www.cnblogs.com/yoyo1216/p/12700989.html

192.168.104.94
192.168.104.96
192.168.104.97

1.创建文件夹

mkdir -p /usr/local/kafka/data  /usr/local/kafka/logs
cd /usr/local/kafka

2.下载Kafka http://kafka.apache.org/downloads

wget http://apache.mirror.iphh.net/kafka/2.5.0/kafka_2.13-2.5.0.tgz

3.解压

tar -zvxf kafka_2.13-2.5.0.tgz

4.修改配置文件

vim /etc/profile
export KAFKA_HOME=/usr/local/kafka/kafka_2.13-2.5.0
export PATH=$KAFKA_HOME/bin:$PATH

source /etc/profile

5.修改配置

cd /usr/local/kafka/kafka_2.13-2.5.0/config
vim server.properties
broker.id=0   # broker id 这个不能重复
listeners=PLAINTEXT://:9092   # 取消注释  默认端口是9092
host.name=0.0.0.0   # ip
log.dirs=/usr/local/kafka/logs  # log文件放置路径
zookeeper.connect=192.168.104.94:2181,192.168.104.96:2181,192.168.104.97:2181   # zookeeper连接信息

6.启动kafka

kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties
kafka-server-start.sh $KAFKA_HOME/config/server.properties
-daemon: 后台启动
config/server.properties: server.properties文件路径
kafka-server-stop.sh  # 关闭kafka

jps  查看进程  jps -m: 查看比较详细的信息
kafka

启动报错: kafka.common.InconsistentClusterIdException: The Cluster ID BmhgroWiQEWltKkCY_4u4Q doesn't match stored clusterId Some(3BVX0S4nTjSvbCvqUU3WGQ) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
    在Kafka config目录中，打开kafka config属性文件，让server.properties查找具有参数log.dirs =的日志路径目录，然后转到日志路径目录并在其中找到文件meta.properties。打开文件meta.properties并更新cluster.id =或从日志路径目录中删除此文件或所有日志文件，然后重新启动kafka。（这个方法并没有解决我的问题）

kafka-server-stop.sh $KAFKA_HOME/config/server.properties  # 关闭kafka

cd /usr/local/kafka/kafka_2.13-2.5.0/logs
cat server.log | tail -n 200  # 查看kafka启动时产生日志

7.创建topic

kafka-topics.sh --zookeeper 192.168.104.94:2181,192.168.104.96:2181,192.168.104.97:2181 --create --topic test-topic --partitions 3 --replication-factor 3  # 创建
    kafka-topics.sh --zookeeper 192.168.104.94:2181,192.168.104.96:2181,192.168.104.97:2181 -list  # 查看所有topic信息
    kafka-topics.sh --zookeeper 192.168.104.94:2181,192.168.104.96:2181,192.168.104.97:2181 --describe --topic test-topic  # 查看指定的topic
    kafka-topics.sh --delete --zookeeper 192.168.104.94:2181,192.168.104.96:2181,192.168.104.97:2181 --topic test-topic  # 删除topic
    kafka-console-consumer.sh --bootstrap-server 192.168.104.94:9092,192.168.104.96:9092,192.168.104.97:9092 --topic test-topic  # 发送消息
    kafka-console-consumer.sh --bootstrap-server 192.168.104.94:9092,192.168.104.96:9092,192.168.104.97:9092 --topic test-topic --from-beginning  # 消费消息