CentOS 7环境下Kafka的安装和基本使用

2017-12-21 09:29 Yang-Onion 阅读(5101) 评论(0) 编辑收藏举报

CentOS 7环境下Kafka的安装和基本使用

基础环境

Windows 10 X64

VMware-workstation-full-12.0.0-2985596

CentOS Linux release 7.4.1708

CentOS 7环境下安装Kafka

安装JDK

在CentOS上将JDK下载到本地.jdk-8u152-linux-x64.tar.gz
解压并将文件移动到/opt/java目录下

sudo tar -vxzf jdk-8u152-linux-x64.tar.gz
sudo mv jdk1.8.0_152 /opt/java

配置java环境变量

vim /etc/profile
#在profile文件的最后面添加
export JAVA_HOME=/opt/jdk1.8.0_152
export PATH=$JAVA_HOME/bin:$PATH
#添加成功后通过java -version来查看jdk是否安装成功
java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

安装Zookeeper

通过wget下载zookeeper到本机

sudo wdget https://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz

解压并将文件移动到/opt/zookeeper目录下

# 解压zookeeper
sudo tar -zxvf zookeeper-3.3.6.tar.gz
# 将zookeeper移动到/opt/zookeeper目录下
sudo mv zookeeper-3.3.6 /opt/zookeeper

编辑zookeeper的配置

# 复制一份zoo_sample.cfg文件并改名为zoo.cfg
sudo cp /opt/zookeeper/zoo_sample.cfg zoo.cfg
# 编辑zoo.cfg 文件
sudo vim /opt/zookeeper/zoo.cfg
#主要修改dataDir和server.1=127.0.0.1:2888:3888这2处
# the directory where the snapshot is stored.
dataDir=/opt/zookeeper/data
# the port at which the clients will connect
clientPort=2181
server.1=127.0.0.1:2888:3888

配置zookeeper环境变量

sudo vim /etc/profile
#添加如下内容
ZOOKEEPER_HOME=/opt/zookeeper
PATH=$PATH:ZOOKEEPER_HOME/bin
export ZOOKEEPER_HOME

启动zookeeper

zkServer.sh start
# 如果上面命令报错,则cd /opt/zookeeper/bin 目录
./zkServer.sh start

安装kafka

通过wget下载kafka到本机

sudo wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/1.0.0/kafka_2.12-1.0.0.tgz

解压并将文件移动到/opt/kafka目录下

# 解压
sudo tar -zxvf kafka_2.12-1.0.0.tgz
# 移动
sudo mv kafka_2.12-1 /opt/kafka

编辑kafka的配置

#创建日志存放目录
cd /opt/kafka
mkdir -p log/kafka
#修改配置文件/opt/kafka/config/server.properties
sudo vim /opt/kafka/config/server.properties
#主要修改下面几项内容如下：
broker.id=181
delete.topic.enable=true
listeners = PLAINTEXT://127.0.0.1:9092
log.dirs=/opt/kafka/log/kafka
zookeeper.connect=127.0.0.1:2181

配置kafka环境变量

sudo vim /etc/profile
#添加如下内容：
KAFKA_HOME=/opt/kafka
PATH=$PATH:$KAFKA_HOME/bin
export PATH KAFKA_HOME

启动kafka

kafka-server-start.sh /opt/kafka/config/server.properties

CentOS 7环境下Kafka的基本使用

创建一个Topic (test)

kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test

Producer向Topic发送消息

kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test

[yang@localhost bin]$ ./kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>producer send one message
>producer send two message
>

Consumer读取Topic的消息

kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test --from-beginning

[yang@localhost bin]$ ./kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 -topic test --from-beginning
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
[2017-11-19 20:32:13,047] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,085] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,100] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,265] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
producer send one message
producer send two message