skills_kafka
skills_kafka
1 offset记录位置
- 首先来说说消费者如果是根据java-api来消费,也就是kafka.javaapi.consumer.ConsumerConnector,通过配置参数zookeeper.connect来消费。这种情况下,消费者的offset会更新到zookeeper的consumers/{group}/offsets/{topic}/{partition}目录下。
[zk: 100.5.14.161:2181(CONNECTED) 20] get /consumers/console-consumer-21030/offsets/topic_test_1/0
37
cZxid = 0x50000053a
ctime = Thu Dec 20 17:26:58 CST 2018
mZxid = 0x50000053a
mtime = Thu Dec 20 17:26:58 CST 2018
pZxid = 0x50000053a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 2
numChildren = 0
- 如果是根据kafka默认的api来消费,即org.apache.kafka.clients.consumer.KafkaConsumer,我们会配置参数bootstrap.servers来消费。而其消费者的offset会更新到一个kafka自带的topic:__consumer_offsets下面,查看当前group的消费进度,则要依靠kafka自带的工具kafka-consumer-offset-checker
[2018-12-25 11:48:10,163] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
Group Topic Pid Offset logSize Lag Owner
test_group_1 topic_test_1 0 489 491 2 none
test_group_1 topic_test_1 1 489 490 1 none
test_group_1 topic_test_1 2 490 491 1 none
小结:kafka的offset即不记录在producer中,也不记录在consumer中。
参考链接:https://blog.csdn.net/lijingjingchn/article/details/85245705