kafka 环境搭建
环境准备
1. zookeeper集群
2. JDK环境
# 解压kafka包 tar xvf kafka_2.11-0.10.1.1.tgz -C /opt/app # cd /opt/app/kafka_2.11-0.10.1.1/ # mkdir logs
修改配置文件
主要关注：server.properties 这个文件即可
kafka有很多文件，这里可以发现有Zookeeper文件，我们可以根据Kafka内带的Zookeeper集群来启动，但是建议使用独立的Zookeeper集群
[root@quan config]# ls -l
total 60
-rw-r--r-- 1 root root 906 Dec 16 2016 connect-console-sink.properties
-rw-r--r-- 1 root root 909 Dec 16 2016 connect-console-source.properties
-rw-r--r-- 1 root root 2760 Dec 16 2016 connect-distributed.properties
-rw-r--r-- 1 root root 883 Dec 16 2016 connect-file-sink.properties
-rw-r--r-- 1 root root 881 Dec 16 2016 connect-file-source.properties
-rw-r--r-- 1 root root 1074 Dec 16 2016 connect-log4j.properties
-rw-r--r-- 1 root root 2061 Dec 16 2016 connect-standalone.properties
-rw-r--r-- 1 root root 1199 Dec 16 2016 consumer.properties
-rw-r--r-- 1 root root 4369 Dec 16 2016 log4j.properties
-rw-r--r-- 1 root root 1900 Dec 16 2016 producer.properties
-rw-r--r-- 1 root root 5336 Dec 16 2016 server.properties
-rw-r--r-- 1 root root 1032 Dec 16 2016 tools-log4j.properties
-rw-r--r-- 1 root root 1023 Dec 16 2016 zookeeper.properties
node 01 -> server.properties
[root@quan config]# egrep -v '#|^$' server.properties # 当前机器在集群中的唯一标识，和zookeeper的myid性质一样 broker.id=0 host.name=192.168.90.2 # 这个是borker进行网络处理的线程数 num.network.threads=3 # 这个是borker进行I/O处理的线程数 num.io.threads=8 # 发送缓冲区buffer大小，数据不是一下子就发送的，先回存储到缓冲区了到达一定的大小后在发送，能提高性能 socket.send.buffer.bytes=102400 # kafka接收缓冲区大小，当数据到达一定大小后在序列化到磁盘 socket.receive.buffer.bytes=102400 # 这个参数是向kafka请求消息或者向kafka发送消息的请请求的最大数，这个值不能超过java的堆栈大小 socket.request.max.bytes=104857600 # 日志目录 log.dirs=/opt/app/kafka_2.11-0.10.1.1/logs # 默认的分区数，一个topic默认1个分区数 num.partitions=1 num.recovery.threads.per.data.dir=1 # 默认消息的最大持久化时间，168小时，7天 log.retention.hours=168 # 因为kafka的消息是以追加的形式落地到文件，当超过这个值的时候，kafka会新起一个文件 log.segment.bytes=1073741824 # 每隔300000毫秒去检查上面配置的log失效时间 log.retention.check.interval.ms=300000 # 设置zookeeper的连接端口 zookeeper.connect=192.168.90.2:2181,192.168.90.3:2181,192.168.90.4:2181 # 设置zookeeper超时时间 zookeeper.connection.timeout.ms=6000 # 配置修改 # 添加以下内容： message.max.byte=5242880 default.replication.factor=2 replica.fetch.max.bytes=5242880 # 修改zookeeper连接端口
启动kafka集群
# 启动
# /opt/app/kafka_2.11-0.10.1.1/bin/kafka-server-start.sh -daemon /opt/app/kafka_2.11-0.10.1.1/config/server.properties # node 01 [root@quan ~]# jps 17575 Jps 789 QuorumPeerMain 17424 Kafka # node 02 [root@quan kafka_2.11-0.10.1.1]# jps 1407 QuorumPeerMain 1727 Kafka 1785 Jps # node 03 [root@quan kafka_2.11-0.10.1.1]# jps 25499 QuorumPeerMain 25875 Jps 25821 Kafka
验证集群
创建topic
[root@quan bin]# ./kafka-topics.sh --create --zookeeper 192.168.90.2:2181 --replication-factor 2 --partitions 1 --topic ftopic Created topic "ftopic". # --replication-factor 2 #复制两份 # --partitions 1 #创建1个分区 # --topic #主题为ftopic # 创建producer # 在node 01 使用console发送消息 [root@quan bin]# ./kafka-console-producer.sh --broker-list 192.168.90.3:9092 --topic ftopic test node 01 -> use --broker-list 192.168.90.3:9092 -----> 192.168.90.2 # 创建consumer # 在node 02 使用console接受消息 [root@quan bin]# ./kafka-console-consumer.sh --zookeeper 192.168.90.2:2181 --topic ftopic --from-beginning Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]. test node 01 -> use --broker-list 192.168.90.3:9092 -----> 192.168.90.2
查询所有topic
# 在node 01 上创建的topic 使用node 02 zookeeper验证 [root@quan bin]# ./kafka-topics.sh --list --zookeeper 192.168.90.3:2181 ftopic # 查看topic状态 [root@quan bin]# ./kafka-topics.sh --describe --zookeeper 192.168.90.3:2181 --topic ftopic Topic:ftopic PartitionCount:1 ReplicationFactor:2 Configs: Topic: ftopic Partition: 0 Leader: 1003 Replicas: 1003,1001 Isr: 1003,1001
使用zookeeper查看kafka三个节点
[root@quan bin]# /opt/app/zookeeper-3.4.6/bin/zkCli.sh -server 192.168.90.3:2181
Connecting to 192.168.90.3:2181
2017-12-12 02:58:23,469 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-12-12 02:58:23,471 [myid:] - INFO [main:Environment@100] - Client environment:host.name=localhost
2017-12-12 02:58:23,471 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_71
2017-12-12 02:58:23,472 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2017-12-12 02:58:23,472 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/app/jdk1.7.0_71/jre
2017-12-12 02:58:23,472 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/app/zookeeper-3.4.6/bin/../build/classes:/opt/app/zookeeper-3.4.6/bin/../build/lib/*.jar:/opt/app/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/app/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/opt/app/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/opt/app/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/opt/app/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/opt/app/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/opt/app/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/opt/app/zookeeper-3.4.6/bin/../conf:.:/opt/app/jdk1.7.0_71/lib/dt.jar:/opt/app/jdk1.7.0_71/lib/tools.jar
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-514.el7.x86_64
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2017-12-12 02:58:23,473 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/app/kafka_2.11-0.10.1.1/bin
2017-12-12 02:58:23,474 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=192.168.90.3:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@75aa57fb
2017-12-12 02:58:23,507 [myid:] - INFO [main-SendThread(192.168.90.3:2181):ClientCnxn$SendThread@975] - Opening socket connection to server 192.168.90.3/192.168.90.3:2181. Will not attempt to authenticate using SASL (unknown error)
2017-12-12 02:58:23,510 [myid:] - INFO [main-SendThread(192.168.90.3:2181):ClientCnxn$SendThread@852] - Socket connection established to 192.168.90.3/192.168.90.3:2181, initiating session
Welcome to ZooKeeper!
2017-12-12 02:58:23,542 [myid:] - INFO [main-SendThread(192.168.90.3:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server 192.168.90.3/192.168.90.3:2181, sessionid = 0x26046a491c00004, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
JLine support is enabled
[zk: 192.168.90.3:2181(CONNECTED) 0] ls /
## 这里 admin consumers config brokers 均为kafka创建
[isr_change_notification, hbase, admin, zookeeper, consumers, cluster, config, controller, pinpoint-cluster, brokers, controller_epoch]
[zk: 192.168.90.3:2181(CONNECTED) 1] ls /brokers
[seqid, topics, ids]
## kafka三个节点, 根据server.properties中broker.id 决定，若将其注释，则kafka会默认分配
[zk: 192.168.90.3:2181(CONNECTED) 2] ls /brokers/ids
[1003, 1001, 1002]
## 查看1001 为 node 02
[zk: 192.168.90.3:2181(CONNECTED) 3] get /brokers/ids/1001
{"jmx_port":-1,"timestamp":"1513016013999","endpoints":["PLAINTEXT://192.168.90.3:9092"],"host":"192.168.90.3","version":3,"port":9092}
cZxid = 0x40000002f
ctime = Tue Dec 12 02:13:44 CST 2017
mZxid = 0x40000002f
mtime = Tue Dec 12 02:13:44 CST 2017
pZxid = 0x40000002f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x36046a4b9cc0000
dataLength = 135
numChildren = 0
## 查看1002 为 node 03
[zk: 192.168.90.3:2181(CONNECTED) 4] get /brokers/ids/1002
{"jmx_port":-1,"timestamp":"1513016023980","endpoints":["PLAINTEXT://192.168.90.4:9092"],"host":"192.168.90.4","version":3,"port":9092}
cZxid = 0x40000002c
ctime = Tue Dec 12 02:13:43 CST 2017
mZxid = 0x40000002c
mtime = Tue Dec 12 02:13:43 CST 2017
pZxid = 0x40000002c
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x36046a4b9cc0001
dataLength = 135
numChildren = 0
## 查看1003 为 node 01
[zk: 192.168.90.3:2181(CONNECTED) 5] get /brokers/ids/1003
{"jmx_port":-1,"timestamp":"1513016051032","endpoints":["PLAINTEXT://192.168.90.2:9092"],"host":"192.168.90.2","version":3,"port":9092}
cZxid = 0x400000034
ctime = Tue Dec 12 02:13:53 CST 2017
mZxid = 0x400000034
mtime = Tue Dec 12 02:13:53 CST 2017
pZxid = 0x400000034
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16046a51e340000
dataLength = 135
numChildren = 0
[zk: 192.168.90.3:2181(CONNECTED) 6]
