Kafka基本使用
使用docker部署kafka
基础环境
| 操作系统 | 硬件配置 |
|---|---|
| CentOS 7 Server | 磁盘:40GB 内存:8GB 网卡:ens3(外网) |
网络配置
# vim ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=192.168.200.13
NETMASK=255.255.255.0
GATEWAY=192.168.200.1
DNS1=114.114.114.114
基础环境配置
安装PIP
# yum install epel-release
# yum install python-pip
# pip install -U pip
安装PIP编译环境
# yum install python-devel libffi-devel gcc openssl-devel
安装docker
# curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh -
# docker --version
配置docker环境
# mkdir -p /etc/systemd/system/docker.service.d
# tee /etc/systemd/system/docker.service.d/kolla.conf <<-'EOF'
[Service]
MountFlags=shared
EOF
### 配置镜像加速
# mkdir -p /etc/docker
# tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://ar4vj99q.mirror.aliyuncs.com"]
}
EOF
# systemctl daemon-reload
# systemctl restart docker
# systemctl enable docker
安装docker-compose
# pip install docker-compose
部署Kafka
下载docker-compose文件
# git clone https://github.com/wurstmeister/kafka-docker.git
# cd kafka-docker
# vim docker-compose.yml
KAFKA_ADVERTISED_HOST_NAME: 192.168.200.13
### 启动kafka
# docker-compose up
### 列举broker
# ./broker-list.sh
:32768
### 如果希望创建多个broker，在执行docker-compose up后执行
# docker-compose scale kafka=3
测试安装
下载kafka
### 测试主机IP： 192.168.200.11
# wget http://mirror.bit.edu.cn/apache/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz -O kafka_2.12-0.10.2.1.tgz
# tar zxf kafka_2.12-0.10.2.1.tgz
# yum install java -y
# cd kafka_2.12-0.10.2.1
创建主题
# bin/kafka-topics.sh --create --zookeeper 192.168.200.13:2181 --replication-factor 1 --partitions 1 --topic fwd-test
# bin/kafka-topics.sh --list --zookeeper 192.168.200.13:2181
### 或者在192.168.200.13执行
# yum install net-tools
# cd kafka-docker
# ./create-topics.sh -n test
测试生产者
### 执行命令后随便输入写字符串并回车结束
# bin/kafka-console-producer.sh --broker-list 192.168.200.13:32768 --topic fwd-test
测试消费者
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.200.13:32768 --topic fwd-test --from-beginning
手工部署kafka
安装Kafka
### 在Sched和Nova节点上执行
# wget http://mirror.bit.edu.cn/apache/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz -O kafka_2.12-0.10.2.1.tgz
# tar zxf kafka_2.12-0.10.2.1.tgz
# yum install java -y
# systemctl disable firewalld
# systemctl stop firewalld
启动Zookeeper
### 在Sched节点上执行
# bin/zookeeper-server-start.sh config/zookeeper.properties &
启动kafka broker
### 在Sched节点上执行
# vim config/server.properties
advertised.listeners=PLAINTEXT://192.168.200.11:9092
# bin/kafka-server-start.sh config/server.properties &
创建topic
### 在Sched节点上执行
# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rsyslog-prod
查看创建的topic
### 在Sched节点上执行
# bin/kafka-topics.sh --list --zookeeper localhost:2181
### 在Nova节点上执行
# bin/kafka-topics.sh --list --zookeeper 192.168.200.11:2181
测试producer
### 在Sched节点上执行，并输入些字符串
# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rsyslog-prod
this server
### 在Nova节点上执行，并输入些字符串
# bin/kafka-console-producer.sh --broker-list 192.168.200.11:9092 --topic rsyslog-prod
this client
测试consumer
### ### 在Sched节点上执行，如果配置无异常则正常显示如下内容
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.200.11:9092 --topic rsyslog-prod --from-beginning
this server
this client
问题处理
kafka-server启动报错
ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.NoClassDefFoundError: Could not initialize class kafka.network.RequestChannel$
at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:93)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.network.Processor.run(SocketServer.scala:421)
at java.lang.Thread.run(Thread.java:745)
### 解决办法，将hostname设置进/etc/hosts
# ip addr show
# hostname
#
