kafka存储结构和查看方式

kafka存储结构和查看方式

参考文档

http://www.taodudu.cc/news/show-4453314.html?action=onClick

https://blog.csdn.net/weixin_42073629/article/details/108906817

1.连接zookeeper

/usr/local/zookeeper-3.4.14/bin/zkCli.sh

ls /: 显示zookeeper根目录下的子节点
有一个节点是：isr_change_notification. 在Kafka 中, Leader 和Follower 的数据同步遵循的是"最终一致"原则, 也就是数据同步会有延迟, 但保证最终数据的一致性.

isr 是'in-sync' replicas 的缩写, 代表的是与Leader 数据已经通过过的replica, 它会作为重选Leader 时作为判断依据.

ls2 /: 显示该节点的子节点信息和属性信息

2.查看brokers

ls /brokers
[ids, topics, seqid]

3.查看有哪些brokers. （1，2，3为server.properties中配置的broker.id值)

ls /brokers/ids
[1, 2, 3]

4.查看broker 1

get /brokers/ids/1
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://10.255.60.141:9092"],"jmx_port":-1,"host":"10.255.60.141","timestamp":"1684826883931","port":9092,"version":4}
说明：
"jmx_port": jmx端口号,当值为-1表示该broker没有设置JMX_PORT
"timestamp": kafka broker初始启动时的时间戳
"host": 主机名或ip地址
"version": 版本编号默认为1
"port": kafka broker的服务端端口号,由server.properties中参数port确定

5.查看controller, 数据显示broker 3 为controller

get /controller
{"version":1,"brokerid":3,"timestamp":"1684826880151"}
说明：
Controller 是一个特殊的Broker, 其负责所有Partition 的leader/follower 关系.
Zookeeper 负责从Broker 中选举出一个作为Controller, 并确保其唯一性. 同时, 当Controller 宕机时, 选举一个新的.

6.查看kafka集群的id

get /cluster/id
{"version":"1","id":"4WVBIm5jSLmINtbQKsH1Ng"}

7.查看有哪些topics

ls /brokers/topics

8.查看topic下有哪些partitions

[zk: localhost:2181(CONNECTED) 1] ls /brokers/topics/__consumer_offsets/partitions
[44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]

9.查看partition 0的状态

get /brokers/topics/__consumer_offsets/partitions/0/state       
{"controller_epoch":59,"leader":1,"version":1,"leader_epoch":68,"isr":[3,2,1]}

10.获取指定topic主题的值

get /brokers/topics/mat_realdata_hcl_line_5min
{"version":2,"partitions":{"2":[3,2,1],"1":[2,1,3],"0":[1,3,2]},"adding_replicas":{},"removing_replicas":{}}
说明：
以"2":[3,2,1]为例，前面的"2"表示partiionId编号，后面的数组表示同步副本组brokerId列表

11.获取指定topic partition的信息

get /brokers/topics/lzdt-itp-trip-pair/partitions/0/state
{"controller_epoch":57,"leader":3,"version":1,"leader_epoch":5,"isr":[3]}
 而且当前状态都是isr(也就是已处于最新状态). 同时, 
说明：
"controller_epoch"：表示kafka集群中的中央控制器选举次数
"leader": 表示该partition选举leader的brokerId，当前的Leader是Broker 3.
"leader_epoch": 该partition leader选举次数
"isr": [同步副本组brokerId列表]
"version": 版本编号默认为1,

12.删除数据

例如：【rmr /brokers/topics/test】：删除test主题

13.consumer注册信息

/consumers/[groupId]/ids/[consumerIdString]
可以ls /consumers查看下