Kafka基本使用

使用docker部署kafka

基础环境

操作系统 硬件配置
CentOS 7 Server 磁盘:40GB 内存:8GB 网卡:ens3(外网)

网络配置

# vim ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=192.168.200.13
NETMASK=255.255.255.0
GATEWAY=192.168.200.1
DNS1=114.114.114.114

基础环境配置

安装PIP

# yum install epel-release
# yum install python-pip
# pip install -U pip

安装PIP编译环境

# yum install python-devel libffi-devel gcc openssl-devel

安装docker

# curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh -
# docker --version

配置docker环境

# mkdir -p /etc/systemd/system/docker.service.d
# tee /etc/systemd/system/docker.service.d/kolla.conf <<-'EOF'
[Service]
MountFlags=shared
EOF

### 配置镜像加速
# mkdir -p /etc/docker
# tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors":  ["https://ar4vj99q.mirror.aliyuncs.com"]
}
EOF
# systemctl daemon-reload
# systemctl restart docker
# systemctl enable docker

安装docker-compose

# pip install docker-compose

部署Kafka

下载docker-compose文件

# git clone https://github.com/wurstmeister/kafka-docker.git
# cd kafka-docker
# vim docker-compose.yml
      KAFKA_ADVERTISED_HOST_NAME: 192.168.200.13
### 启动kafka
# docker-compose up
### 列举broker
# ./broker-list.sh
:32768

### 如果希望创建多个broker,在执行docker-compose up后执行
# docker-compose scale kafka=3

测试安装

下载kafka

### 测试主机IP: 192.168.200.11
# wget http://mirror.bit.edu.cn/apache/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz -O kafka_2.12-0.10.2.1.tgz
# tar zxf kafka_2.12-0.10.2.1.tgz
# yum install java -y
# cd kafka_2.12-0.10.2.1

创建主题

# bin/kafka-topics.sh --create --zookeeper 192.168.200.13:2181 --replication-factor 1 --partitions 1 --topic fwd-test
# bin/kafka-topics.sh --list --zookeeper 192.168.200.13:2181

### 或者在192.168.200.13执行
# yum install net-tools
# cd kafka-docker
# ./create-topics.sh -n test

测试生产者

### 执行命令后随便输入写字符串并回车结束
# bin/kafka-console-producer.sh --broker-list 192.168.200.13:32768 --topic fwd-test

测试消费者

# bin/kafka-console-consumer.sh --bootstrap-server 192.168.200.13:32768 --topic fwd-test --from-beginning

手工部署kafka

安装Kafka

### 在Sched和Nova节点上执行
# wget http://mirror.bit.edu.cn/apache/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz -O kafka_2.12-0.10.2.1.tgz
# tar zxf kafka_2.12-0.10.2.1.tgz
# yum install java -y
# systemctl disable firewalld
# systemctl stop firewalld

启动Zookeeper

### 在Sched节点上执行
# bin/zookeeper-server-start.sh config/zookeeper.properties &

启动kafka broker

### 在Sched节点上执行
# vim config/server.properties
advertised.listeners=PLAINTEXT://192.168.200.11:9092
# bin/kafka-server-start.sh config/server.properties &

创建topic

### 在Sched节点上执行
# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rsyslog-prod

查看创建的topic

### 在Sched节点上执行
# bin/kafka-topics.sh --list --zookeeper localhost:2181

### 在Nova节点上执行
# bin/kafka-topics.sh --list --zookeeper 192.168.200.11:2181

测试producer

### 在Sched节点上执行,并输入些字符串
# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rsyslog-prod
this server

### 在Nova节点上执行,并输入些字符串
# bin/kafka-console-producer.sh --broker-list 192.168.200.11:9092 --topic rsyslog-prod
this client

测试consumer

### ### 在Sched节点上执行,如果配置无异常则正常显示如下内容
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.200.11:9092 --topic rsyslog-prod --from-beginning
this server
this client

问题处理

kafka-server启动报错

ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.NoClassDefFoundError: Could not initialize class kafka.network.RequestChannel$
    at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:93)
    at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
    at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
    at scala.collection.Iterator$class.foreach(Iterator.scala:742)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at kafka.network.Processor.run(SocketServer.scala:421)
    at java.lang.Thread.run(Thread.java:745)

### 解决办法,将hostname设置进/etc/hosts
# ip addr show
# hostname
# 

参考文档

Landoop GitHub

Confluent官方文档

posted @ 2017-10-14 11:52  银魔术师  阅读(1572)  评论(0编辑  收藏  举报