kafka总结

1、启动kafka

sh kafka-server-start.sh ../config/server.properties &

注意：如果报Exception in thread "main" java.net.UnknownHostException错误，是因为kafka获取ip信息的时候使用的是java.net.InetAddress.getCanonicalHostName()，在linux下该方法获取的是主机名。

 

说明：broker注册到zookeeper的IP是用的server.properties配置的信息，查看server.properties的注释:（配置了，使用advertised.host.name的信息，不然使用host.name的信息，如果Host.name也是空，则使用java.net.InetAddress.getCanonicalHostName()）　

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=xxx.xxx.xxx.xxx

# Hostname the broker will advertise to producers and consumers. If not set, it uses the
# value for "host.name" if configured.  Otherwise, it will use the value returned from
# java.net.InetAddress.getCanonicalHostName().
advertised.host.name=xxx.xxx.xxx.xxx

 

解决办法：

　　1>、host里配置：localhost 主机名。

　　2>、配置kafka 的server.properties中，host.name=本机IP（一般是内网IP）　　

　　如果使用方式1，在本机启动producer是OK的，但是在远程就不行了，因为返回给client的是zookeeper里注册的节点信息localhost，client通过localhost肯定是获取不到broker地址的。所以此处直接配置内网IP即可解决

另外，在linux中，consumer启动的时候，注册到zookeeper的IP信息应该也是通过java.net.InetAddress.getCanonicalHostName()获取的，所以，在linux上启动consumer需要配置host

 

2、zookeeper上查看节点信息 

sh zkCli.sh -server 127.0.0.1:2181
ls /brokers/ids

 

3、创建topic（2个分区，3个副本）

./kafka-topics.sh --create --topic topic1 --replication-factor 3 --partitions 2 --zookeeper localhost:2181

4、查看topic list

 ./kafka-topics.sh --list --zookeeper localhost:2181  

5、zookeeper上查看topic list信息

　 ls /brokers/topics

6、zookeeper上查看节点具体信息

[zk: 127.0.0.1:2181(CONNECTED) 9] get /brokers/topics/topic1
{"version":1,"partitions":{"1":[1,2,3],"0":[3,1,2]}}
cZxid = 0x200000179
ctime = Wed Mar 30 23:24:25 EDT 2016
mZxid = 0x200000179
mtime = Wed Mar 30 23:24:25 EDT 2016
pZxid = 0x20000017d
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 52
numChildren = 1

7、kafka目录中查看节点信息

cd /tmp/kafka-logs-1 #kafka server.config中配置的目录，topic1-0 和topic1-1分别记录topic1的2个分区的日志数据。查看另外2个kafka日志目录，也会存在这样的2个文件夹，对应的是topic创建的3个副本
drwxr-xr-x 2 root root 4096 Mar 30 23:24 topic1-0
drwxr-xr-x 2 root root 4096 Mar 30 23:24 topic1-1

8、创建消费者

 ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic topic1 --from-beginning

9、zookeeper中查看对应的节点信息

#自动创建group_id
[zk: 127.0.0.1:2181(CONNECTED) 18] ls /consumers
[console-consumer-75348]
#对应三个节点
[zk: 127.0.0.1:2181(CONNECTED) 19] ls /consumers/console-consumer-75348
[offsets, owners, ids]
#查看刚刚创建的消费者ID：consumerId，用来标记一个消费者
[zk: 127.0.0.1:2181(CONNECTED) 15] ls /consumers/console-consumer-75348/ids
[console-consumer-75348_e2ef967443a3-1459395856932-a0028f17]
[zk: 127.0.0.1:2181(CONNECTED) 17] get /consumers/console-consumer-75348/ids/console-consumer-75348_e2ef967443a3-1459395856932-a0028f17
{"version":1,"subscription":{"topic1":1},"pattern":"white_list","timestamp":"1459395857222"}
cZxid = 0x200000190
ctime = Wed Mar 30 23:44:17 EDT 2016
mZxid = 0x200000190
mtime = Wed Mar 30 23:44:17 EDT 2016
pZxid = 0x200000190
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x153413bc26e002d
dataLength = 92
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 20] ls /consumers/console-consumer-75348/owners
[topic1]
[zk: 127.0.0.1:2181(CONNECTED) 21] ls /consumers/console-consumer-75348/owners/topic1
[1, 0]
#topic1的第一个分区正被xxx消费
[zk: 127.0.0.1:2181(CONNECTED) 23] get /consumers/console-consumer-75348/owners/topic1/1
console-consumer-75348_e2ef967443a3-1459395856932-a0028f17-0
cZxid = 0x200000195
ctime = Wed Mar 30 23:44:18 EDT 2016
mZxid = 0x200000195
mtime = Wed Mar 30 23:44:18 EDT 2016
pZxid = 0x200000195
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x153413bc26e002d
dataLength = 60
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 24] get /consumers/console-consumer-75348/owners/topic1/0
console-consumer-75348_e2ef967443a3-1459395856932-a0028f17-0
cZxid = 0x200000196
ctime = Wed Mar 30 23:44:18 EDT 2016
mZxid = 0x200000196
mtime = Wed Mar 30 23:44:18 EDT 2016
pZxid = 0x200000196
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x153413bc26e002d
dataLength = 60
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 26] ls /consumers/console-consumer-75348/offsets 
[topic1]
[zk: 127.0.0.1:2181(CONNECTED) 27] ls /consumers/console-consumer-75348/offsets/topic1
[1, 0]
#topic1的1分区被消费的offset位置。和consumer_id无关，和group_id关联
#就是说，每个分区的信息可以被不同的group_id消费，但是同一时间只能由一个group的其中一个consumer消费
[zk: 127.0.0.1:2181(CONNECTED) 29] get /consumers/console-consumer-75348/offsets/topic1/1
0
cZxid = 0x20000019e
ctime = Wed Mar 30 23:45:17 EDT 2016
mZxid = 0x20000019e
mtime = Wed Mar 30 23:45:17 EDT 2016
pZxid = 0x20000019e
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 30] get /consumers/console-consumer-75348/offsets/topic1/0
0
cZxid = 0x20000019b
ctime = Wed Mar 30 23:45:17 EDT 2016
mZxid = 0x20000019b
mtime = Wed Mar 30 23:45:17 EDT 2016
pZxid = 0x20000019b
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0

10、创建生产者

sh kafka-console-producer.sh --broker-list localhost:9092  --topic topic1

11、随便输入消息，可以在消费者控制台看到对应的信息。

12、zk上查看对应的信息

[zk: localhost:2181(CONNECTED) 42] get /consumers/console-consumer-19167/offsets/topic1/0
3
cZxid = 0x2000001fc
ctime = Thu Mar 31 01:50:52 EDT 2016
mZxid = 0x200000204
mtime = Thu Mar 31 01:52:51 EDT 2016
pZxid = 0x2000001fc
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0

可以看到offset有变化

好了，初步进行了总结，还是多动手才能有更深刻的体会