Nginx整合kafka进行高QPS的消息接收

先用本地虚拟机进行测试搭建整体环境。

一、环境以及软件版本

环境	版本
操作系统：Centos 7 x86_64	7
虚拟机：VMware	12.5
远程连接：XShell	5.0
kafka	kafka_2.13-2.6.1
nginx	nginx-1.17.8
zookeeper	zookeeper-3.4.14

二、zookeeper安装

```sh
wget http://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar -zxf zookeeper-3.4.14.tar.gz
cd /home/wyh/data/software/zookeeper-3.4.14/conf
cp zoo_sample.cfg zoo.cfg
# 编辑zoo.cfg文件
vim zoo.cfg
#修改zookeeper的dataDir
dataDir=/home/wyh/data/zookeeper/data
```

编辑/etc/profile

```shell
vim /etc/profile
# ZOOKEEPER_PREFIX指向Zookeeper的解压目录
export ZOOKEEPER_PROFIX=/home/wyh/data/software/zookeeper-3.4.14
# 将Zookeeper的bin目录添加到PATH中
export PATH=$PATH:$ZOOKEEPER_PROFIX/bin
# 设置环境变量ZOO_LOG_DIR，指定Zookeeper保存日志的位置
export ZOO_LOG_DIR=/home/wyh/data/zookeeper/log
#使配置生效
source /etc/profile
#验证
zkServer.sh status
```

三、kafka安装

1、软件下载

wget http://mirrors.hust.edu.cn/apache/kafka/2.6.1/kafka_2.13-2.6.1.tgz
tar xvf kafka_2.13-2.6.1.tgz 

2、配置环境变量

vim /etc/profile
​
export KAFKA_HOME=/home/wyh/data/software/kafka_2.13-2.6.1
export PATH=$PATH:$KAFKA_HOME/bin
​
source /etc/profile

3、配置kafka 连接Zookeeper地址

vim /home/wyh/data/software/kafka_2.13-2.6.1/config/server.properties
​
log.dirs=/home/wyh/data/kafka-logs
zookeeper.connect=localhost:2181/myKafka

4、启动zookeeper和kafka

#启动zookeeper
zkServer.sh start
#启动kafka
bin/kafka-server-start.sh   config/server.properties

四、安装nginx

#下载nginx
wget http://nginx.org/download/nginx-1.17.8.tar.gz
tar -zxf nginx-1.17.8.tar.gz
​
#下载kafka的c客户端代码
git clone https://github.com/edenhill/librdkafka
​
# 进入到librdkafka，然后进行编译
cd librdkafka
yum install -y gcc gcc-c++ pcre-devel zlib-devel
./configure
make && make install
​
# 安装nginx整合kafka的插件
git clone https://github.com/brg-liuwei/ngx_kafka_module.git
cd nginx-1.17.8/
./configure --add-module=/home/wyh/data/software/ngx_kafka_module
make && make install

修改nginx的配置文件

cd /usr/local/nginx/conf
vim nginx.conf
#http块里添加
kafka;
kafka_broker_list 192.168.44.3:9092;
#server块里添加
location /kafka/log {
    kafka_topic top_log1;
}
# 让操作系统加载模板
echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig
​
# 启动nginx
/usr/local/nginx/sbin/nginx

五、测试消息发送

#发送消息
curl 192.168.44.3:8001/kafka/log -d "message send to kafka top_log1"
#获取消息
[root@vm bin]# kafka-console-consumer.sh --bootstrap-server 192.168.44.3:9092 --topic top_log1 --from-beginning
message send to kafka top_log1