ELK03-使用kafka作为缓存收集日志
1.Kafka简介
Kafka 被称为下一代分布式消息系统，是非营利性组织ASF(Apache Software Foundation，简称为ASF)基金会中的一个开源项目，比如HTTP Server、Hadoop、ActiveMQ、Tomcat等开源软件都属于Apache基金会的开源软件，类似的消息系统还有RbbitMQ、ActiveMQ、ZeroMQ，最主要的优势是其具备分布式功能、并且结合zookeeper可以实现动态扩容。
2.部署和配置Kafka集群
1）安装Kafka集群前的准备工作
三台Kafka需要互相能够ping通，所以需要配置/etc/hosts文件，我自己测试机配置如下：
第1台主机：
[root@elk01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.96.211.209 elk01
10.96.211.110 elk02
10.96.211.111 elk03
[root@elk01 ~]# ping elk02
PING elk02 (10.96.211.110) 56(84) bytes of data.
64 bytes from elk02 (10.96.211.110): icmp_seq=1 ttl=64 time=0.802 ms
64 bytes from elk02 (10.96.211.110): icmp_seq=2 ttl=64 time=0.389 ms
^C
--- elk02 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.389/0.595/0.802/0.207 ms
You have new mail in /var/spool/mail/root
[root@elk01 ~]# ping elk03
PING elk03 (10.96.211.111) 56(84) bytes of data.
64 bytes from elk03 (10.96.211.111): icmp_seq=1 ttl=64 time=0.942 ms
64 bytes from elk03 (10.96.211.111): icmp_seq=2 ttl=64 time=1.05 ms
^C
--- elk03 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.942/0.996/1.051/0.062 ms
第2台主机：
[root@elk02 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.96.211.110 elk02
10.96.211.209 elk01
10.96.211.111 elk03
[root@elk02 ~]# ping elk01
PING elk01 (10.96.211.209) 56(84) bytes of data.
64 bytes from elk01 (10.96.211.209): icmp_seq=1 ttl=64 time=0.413 ms
64 bytes from elk01 (10.96.211.209): icmp_seq=2 ttl=64 time=0.416 ms
^C
--- elk01 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1039ms
rtt min/avg/max/mdev = 0.413/0.414/0.416/0.020 ms
[root@elk02 ~]# ping elk03
PING elk03 (10.96.211.111) 56(84) bytes of data.
64 bytes from elk03 (10.96.211.111): icmp_seq=1 ttl=64 time=0.450 ms
64 bytes from elk03 (10.96.211.111): icmp_seq=2 ttl=64 time=0.807 ms
64 bytes from elk03 (10.96.211.111): icmp_seq=3 ttl=64 time=0.501 ms
^C
--- elk03 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2107ms
rtt min/avg/max/mdev = 0.450/0.586/0.807/0.157 ms
第3台主机：
[root@elk03 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.96.211.110 elk02
10.96.211.209 elk01
10.96.211.111 elk03
[root@elk03 ~]# ping elk01
PING elk01 (10.96.211.209) 56(84) bytes of data.
64 bytes from elk01 (10.96.211.209): icmp_seq=1 ttl=64 time=0.865 ms
64 bytes from elk01 (10.96.211.209): icmp_seq=2 ttl=64 time=0.971 ms
^C
--- elk01 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.865/0.918/0.971/0.053 ms
[root@elk03 ~]# ping elk02
PING elk02 (10.96.211.110) 56(84) bytes of data.
64 bytes from elk02 (10.96.211.110): icmp_seq=1 ttl=64 time=0.455 ms
64 bytes from elk02 (10.96.211.110): icmp_seq=2 ttl=64 time=0.660 ms
^C
--- elk02 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.455/0.557/0.660/0.105 ms
2）安装标准3台Kafka集群
a. zookeeper简介
Kafka安装使用依赖zookeeper,所以安装Kafka之前需要安装zookeeper。
zookeeper集群特性:整个集群中只要有超过集群数量一半的zookeeper工作是正常的,那么整个集群对外就是可用的,例如有2台服务器做了一个zaookeeper,只要有任何一台故障或宕机,那么这个zookeeper集群就是不可用的了.因为剩下的一台没有超过集群的一半的数量,但是假如有三台zookeeper组成一个集群,那么损坏一台还剩两台,大于3台的一半,所以损坏一台还是可以正常运行的,但是再损坏一台就只剩下一台,集群就不可用了。
如果是4台组成,损坏一台正常,损坏两台还剩两台,不满足集群总数的一半,所以3台的集群和4台的集群算坏两台的结果都是集群不可用.所以这也是为什么集群一般是奇数的原因。
b. 安装配置zookeeper
Kafka安装使用依赖zookeeper,所以安装Kafka之前需要安装zookeeper。
安装zookeeper前的准备工作
安装JDK1.8，如果主机默认存在JDK1.8，就不需要单独安装JDK了。我的三台主机默认是JDK1.8我就不安装了。
三台主机创建统一的存放软件包的目录：/opt/software
[root@elk01 ~]# mkdir /opt/software
[root@elk02 ~]# mkdir /opt/software
[root@elk03 ~]# mkdir /opt/software
安装zookeeper过程如下：
第1台主机：
[root@elk01 software]# pwd
/opt/software
[root@elk01 software]# [root@elk01 software]# tar zxf zookeeper-3.4.11.tar.gz -C /opt/
[root@elk01 software]# ln -s /opt/zookeeper-3.4.11/ /opt/zookeeper
[root@elk01 software]# mkdir -p /data/zookeeper
[root@elk01 software]# cp /opt/zookeeper/conf/zoo_sample.cfg /opt/zookeeper/conf/zoo.cfg
[root@elk01 software]# grep "^[a-Z]" /opt/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
server.1=10.96.211.209:2888:3888
server.2=10.96.211.110:2888:3888
server.3=10.96.211.111:2888:3888
[root@elk01 software]# echo "1" > /data/zookeeper/myid
[root@elk01 software]# cat /data/zookeeper/myid
1
第2台主机和第3台主机：
在第1台主机上进行操作，直接拷贝给第2台和第3台主机
[root@elk01 opt]# scp -r zookeeper* elk02:/opt/
[root@elk01 opt]# scp -r zookeeper* elk03:/opt/
然后第2台主机在做如下操作：
[root@elk02 software]# echo "2" > /data/zookeeper/myid
[root@elk02 software]# cat /data/zookeeper/myid
2
最后第3台主机在做如下操作：
[root@elk03 software]# echo "3" > /data/zookeeper/myid
[root@elk03 software]# cat /data/zookeeper/myid
3
c. 各节点启动zookeeper
节点1：
[root@elk01 software]# /opt/zookeeper/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
节点2：
[root@elk02 software]# /opt/zookeeper/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
节点3：
[root@elk03 software]# /opt/zookeeper/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
d. 查看各个节点的zookeeper状态
节点1
[root@elk01 opt]# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: follower
节点2
[root@elk02 zookeeper]# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: follower
节点3
[root@elk03 zookeeper]# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: leader
e. zookeeper基本操作
连接到任意节点生成数据:我们在节点1生成数据,然后在其他节点查看验证数据：
[root@elk01 opt]# /opt/zookeeper/bin/zkCli.sh -server elk01:2181
Connecting to elk01:2181
2022-02-11 15:04:57,730 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
2022-02-11 15:04:57,732 [myid:] - INFO [main:Environment@100] - Client environment:host.name=elk01
2022-02-11 15:04:57,732 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_312
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Red Hat, Inc.
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper/bin/../zookeeper-3.4.11.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-1160.el7.x86_64
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2022-02-11 15:04:57,734 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt
2022-02-11 15:04:57,735 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=elk01:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
2022-02-11 15:04:57,747 [myid:] - INFO [main-SendThread(elk01:2181):ClientCnxn$SendThread@1035] - Opening socket connection to server elk01/10.96.211.209:2181. Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
JLine support is enabled
2022-02-11 15:04:57,788 [myid:] - INFO [main-SendThread(elk01:2181):ClientCnxn$SendThread@877] - Socket connection established to elk01/10.96.211.209:2181, initiating session
2022-02-11 15:04:57,815 [myid:] - INFO [main-SendThread(elk01:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server elk01/10.96.211.209:2181, sessionid = 0x100015eb5d20000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: elk01:2181(CONNECTED) 0] create /test "hello"
Created /test
在其他节点上查看验证数据：连接节点2 zookeeper查看
[root@elk02 zookeeper]# /opt/zookeeper/bin/zkCli.sh -server elk02:2181
Connecting to elk02:2181
2022-02-11 15:06:09,783 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
2022-02-11 15:06:09,786 [myid:] - INFO [main:Environment@100] - Client environment:host.name=elk02
2022-02-11 15:06:09,786 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_312
2022-02-11 15:06:09,789 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Red Hat, Inc.
2022-02-11 15:06:09,789 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper/bin/../zookeeper-3.4.11.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-1160.el7.x86_64
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2022-02-11 15:06:09,790 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/zookeeper
2022-02-11 15:06:09,791 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=elk02:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
2022-02-11 15:06:09,810 [myid:] - INFO [main-SendThread(elk02:2181):ClientCnxn$SendThread@1035] - Opening socket connection to server elk02/10.96.211.110:2181. Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
JLine support is enabled
2022-02-11 15:06:09,865 [myid:] - INFO [main-SendThread(elk02:2181):ClientCnxn$SendThread@877] - Socket connection established to elk02/10.96.211.110:2181, initiating session
2022-02-11 15:06:09,887 [myid:] - INFO [main-SendThread(elk02:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server elk02/10.96.211.110:2181, sessionid = 0x200015cabf00000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: elk02:2181(CONNECTED) 0] get /test
hello
cZxid = 0x100000002
ctime = Fri Feb 11 15:05:27 CST 2022
mZxid = 0x100000002
mtime = Fri Feb 11 15:05:27 CST 2022
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 5
numChildren = 0
在其他节点上查看验证数据：连接节点3 zookeeper查看
[root@elk03 zookeeper]# /opt/zookeeper/bin/zkCli.sh -server elk03:2181
Connecting to elk03:2181
2022-02-11 15:07:08,968 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
2022-02-11 15:07:08,970 [myid:] - INFO [main:Environment@100] - Client environment:host.name=elk03
2022-02-11 15:07:08,970 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_312
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Red Hat, Inc.
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper/bin/../zookeeper-3.4.11.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-1160.el7.x86_64
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2022-02-11 15:07:08,972 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/zookeeper
2022-02-11 15:07:08,973 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=elk03:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
Welcome to ZooKeeper!
2022-02-11 15:07:08,987 [myid:] - INFO [main-SendThread(elk03:2181):ClientCnxn$SendThread@1035] - Opening socket connection to server elk03/10.96.211.111:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2022-02-11 15:07:09,027 [myid:] - INFO [main-SendThread(elk03:2181):ClientCnxn$SendThread@877] - Socket connection established to elk03/10.96.211.111:2181, initiating session
2022-02-11 15:07:09,040 [myid:] - INFO [main-SendThread(elk03:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server elk03/10.96.211.111:2181, sessionid = 0x300015c8faa0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: elk03:2181(CONNECTED) 0] get /test
hello
cZxid = 0x100000002
ctime = Fri Feb 11 15:05:27 CST 2022
mZxid = 0x100000002
mtime = Fri Feb 11 15:05:27 CST 2022
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 5
numChildren = 0
f. 安装配置Kafka集群
安装3台Kafka集群
第1台主机：
[root@elk01 software]# pwd
/opt/software
[root@elk01 software]# tar zxf kafka_2.11-1.0.0.tgz -C /opt/
[root@elk01 software]# ln -s /opt/kafka_2.11-1.0.0/ /opt/kafka
[root@elk01 software]# mkdir /opt/kafka/logs
[root@elk01 software]# cat /opt/kafka/config/server.properties
broker.id=1
listeners=PLAINTEXT://10.96.211.209:9092
log.dirs=/opt/kafka/logs
log.retention.hours=24
zookeeper.connect=10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181
第2台和第3台主机：
在第1台主机进行操作，直接解压后的kafka目录 拷贝给第2台和第3台主机
[root@elk01 opt]# pwd
/opt
[root@elk01 opt]# scp -r kafka* elk02:/opt/
[root@elk01 opt]# scp -r kafka* elk03:/opt/
修改第2台主机有关配置文件,创建日志目录
[root@elk02 opt]# cat /opt/kafka/config/server.properties
broker.id=2
listeners=PLAINTEXT://10.96.211.110:9092
log.dirs=/opt/kafka/logs
log.retention.hours=24
zookeeper.connect=10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181
[root@elk02 opt]# mkdir /opt/kafka/logs
修改第3台主机有关配置文件,创建日志目录
[root@elk03 zookeeper]# cat /opt/kafka/config/server.properties
broker.id=3
listeners=PLAINTEXT://10.96.211.111:9092
log.dirs=/opt/kafka/logs
log.retention.hours=24
zookeeper.connect=10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181
[root@elk03 zookeeper]# mkdir /opt/kafka/logs
各节点启动kafka
节点1
[root@elk01 opt]# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
[root@elk01 opt]# tail -f /opt/kafka/logs/server.log
[2022-02-11 16:18:25,168] INFO [ProducerId Manager 1]: Acquired new producerId block (brokerId:1,blockStartProducerId:1000,blockEndProducerId:1999) by writing to Zk with path version 2 (kafka.coordinator.transaction.ProducerIdManager)
[2022-02-11 16:18:25,178] INFO [TransactionCoordinator id=1] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-02-11 16:18:25,179] INFO [TransactionCoordinator id=1] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-02-11 16:18:25,182] INFO [Transaction Marker Channel Manager 1]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-02-11 16:18:25,240] INFO Creating /brokers/ids/1 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2022-02-11 16:18:25,248] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2022-02-11 16:18:25,249] INFO Registered broker 1 at path /brokers/ids/1 with addresses: EndPoint(10.96.211.209,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2022-02-11 16:18:25,257] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-11 16:18:25,257] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-11 16:18:25,259] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)
节点2
[root@elk02 opt]# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
[root@elk02 opt]# tail -f /opt/kafka/logs/server.log
[2022-02-11 16:24:06,305] INFO [TransactionCoordinator id=2] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-02-11 16:24:06,309] INFO [Transaction Marker Channel Manager 2]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-02-11 16:24:06,309] INFO [TransactionCoordinator id=2] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-02-11 16:24:06,404] INFO Creating /brokers/ids/2 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2022-02-11 16:24:06,424] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2022-02-11 16:24:06,427] INFO Registered broker 2 at path /brokers/ids/2 with addresses: EndPoint(10.96.211.110,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2022-02-11 16:24:06,435] WARN No meta.properties file under dir /opt/kafka/logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2022-02-11 16:24:06,485] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-11 16:24:06,485] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-11 16:24:06,489] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)
节点3
[root@elk03 zookeeper]# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
You have new mail in /var/spool/mail/root
[root@elk03 zookeeper]# tail -f /opt/kafka/logs/server.log
[2022-02-11 16:25:11,076] INFO [TransactionCoordinator id=3] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-02-11 16:25:11,077] INFO [Transaction Marker Channel Manager 3]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-02-11 16:25:11,077] INFO [TransactionCoordinator id=3] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-02-11 16:25:11,105] INFO Creating /brokers/ids/3 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2022-02-11 16:25:11,115] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2022-02-11 16:25:11,116] INFO Registered broker 3 at path /brokers/ids/3 with addresses: EndPoint(10.96.211.111,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2022-02-11 16:25:11,118] WARN No meta.properties file under dir /opt/kafka/logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2022-02-11 16:25:11,143] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-11 16:25:11,144] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-11 16:25:11,145] INFO [KafkaServer id=3] started (kafka.server.KafkaServer)
测试验证kafka
创建名为kafkatest，partitions(分区)为3，replication(复制)为3的topic(主题),在任意机器操作都可以。
我在节点1测试创建：
[root@elk01 opt]# /opt/kafka/bin/kafka-topics.sh --create --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --partitions 3 --replication-factor 3 --topic kafkatest
Created topic "kafkatest".
可以在任意1台主机进行测试
节点1:
[root@elk01 opt]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic kafkatest Topic:kafkatest PartitionCount:3 ReplicationFactor:3 Configs:
Topic: kafkatest Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: kafkatest Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: kafkatest Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
节点2：
[root@elk02 opt]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic kafkatest Topic:kafkatest
Topic:kafkatest PartitionCount:3 ReplicationFactor:3 Configs:
Topic: kafkatest Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: kafkatest Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: kafkatest Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
节点3：
[root@elk03 zookeeper]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic kafkatest Topic:kafkatest
Topic:kafkatest PartitionCount:3 ReplicationFactor:3 Configs:
Topic: kafkatest Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: kafkatest Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: kafkatest Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
kafka测试发送消息
kafka测试发送消息:在节点1发消息，在其它节点都能收到
节点1：
[root@elk01 opt]# /opt/kafka/bin/kafka-topics.sh --create --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --partitions 3 --replication-factor 3 --topic messagetest
Created topic "messagetest".
[root@elk01 opt]# /opt/kafka/bin/kafka-console-producer.sh --broker-list 10.96.211.209:9092,10.96.211.110:9092,10.96.211.111:9092 --topic messagetest
>hello
>ron
>
节点2：
[root@elk02 opt]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic messagetest --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
hello
ron
节点3：
[root@elk03 zookeeper]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic messagetest --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
hello
ron
节点2和节点3退出重新登录，还能重新看到收到历史消息记录：
[root@elk02 opt]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic messagetest --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
ron
hello
[root@elk03 zookeeper]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic messagetest --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
hello
ron
^CProcessed a total of 2 messages
You have new mail in /var/spool/mail/root
[root@elk03 zookeeper]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.96.211.209:2181,10.96.211.110:2181,10.96.211.111:2181 --topic messagetest --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
ron
hello
3.使用kafka作为缓存存储日志
1)配置filebeat
filebeat配置文件配置后的内容如下：
[root@elk01 logs]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
setup.kibana:
host: "10.96.211.209:5601"
output.kafka:
hosts: ["10.96.211.209:9092","10.96.211.110:9092","10.96.211.111:9092"]
topic: elklog
2)安装和配置logstash
[root@elk01 es_soft]# rpm -ivh logstash-6.6.0.rpm
logstash配置文件不需要修改，主配置文件在/etc/logstash/conf.d里。
[root@elk01 logs]# cat /etc/logstash/conf.d/kafka.conf
input{
kafka{
bootstrap_servers=>"10.96.211.209:9092"
topics=>["elklog"]
group_id=>"logstash"
codec => "json"
}
}
filter {
mutate {
convert => ["upstream_time", "float"]
convert => ["request_time", "float"]
}
}
output {
if "access" in [tags] {
elasticsearch {
hosts => "http://localhost:9200"
manage_template => false
index => "nginx_access-%{+yyyy.MM}"
}
}
if "error" in [tags] {
elasticsearch {
hosts => "http://localhost:9200"
manage_template => false
index => "nginx_error-%{+yyyy.MM}"
}
}
}
3)启动logstash
[root@elk01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka.conf
4）重启filebeat
systemctl restart filebeat
5）模拟访问（造nginx日志）
ab -n 20000 -c 20 http://10.96.211.209/0214testhaha
6)查看kibana或者有关日志
