Kafka 终端操作
老版本Kafka（zk)
创建topic
kafka-topics -create --zookeeper 192.168.1.11:2181 --replication-factor 1 --partitions 1 --topic zainfo-xm an-baseline-web-20200716142924
kafka-topics -create --zookeeper 192.168.1.11:2181/zaif_dev_kafka --replication-factor 1 --partitions 1 --topic xman-baseline-reach-push-20210421175335
查看topic列表：
kafka-topics --zookeeper 192.168.1.11:2181 --list
使用zk查看topic
#连接zk
zkc z-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:2181
Connecting to z-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:2181
.....
# 查看topic
[zk: z-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:2181(CONNECTED) 2] ls /brokers/topics |grep test-topic
'ls path [watch]' has been deprecated. Please use 'ls [-w] path' instead.
[TLSTestTopic, __amazon_msk_canary, __amazon_msk_canary_state, __consumer_command_request, __consumer_offsets, __consumer_offsets-companjon-meta002, audit-aia-dev1, audit-aia-dev2, audit-aia-dev3, audit-aia-test1, audit-aia-test2, audit-aia-test3, audit-anan-cfg, audit-companjon-cfg, audit-companjon-dev, audit-companjon-dev2, audit-companjon-test,
通过zk删除topic (未测)
rmr /brokers/topics/audit-aia-test3
新版本Kafka
创建topic
kafka-topics --create --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --replication-factor 2 --partitions 3 --topic xman-baseline-reach-sms-incomenew-test1
Created topic xman-baseline-reach-sms-incomenew-test1.
#--replication-factor：topic的总副本数，1代表主备节点总共只有一个副本，相当于没有备份，所以通常3节点的话设置为2 #--partitions：topic分区的数量，通常和节点数保持一致 #--topic：队列名字
查看topic列表：
kafka-topics --list --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 |grep xman-baseline-reach-sms-incomenew-test1
通过zk查看topic:(超慢)
kafka-topics --zookeeper z-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:2181 --list |grep xman-baseline-reach-sms-incomenew-test1
xman-baseline-reach-sms-incomenew-test1
查看topic详细信息：
kafka-topics --describe --topic xman-baseline-reach-sms-incomenew-test1 --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092
Topic: xman-baseline-reach-sms-incomenew-test1 PartitionCount: 3 ReplicationFactor: 2 Configs: min.insync.replicas=2,message.format.version=2.2-IV1,unclean.leader.election.enable=true
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 0 Leader: 1 Replicas: 1,3 Isr: 1,3
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 1 Leader: 3 Replicas: 3,2 Isr: 3,2
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 2 Leader: 2 Replicas: 2,1 Isr: 2,1
Partition是分区编号，Leader，Replicats和Isr里是broker_id，而broker_id是在$KAFKA_HOME/config/server.properties里配置的
查看topic内容
kafka-console-consumer --bootstrap-server b-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --topic xman-baseline-cms-api-kafka-xman-genesis-sit --from-beginning
this is a test
跟查看消费一样
删除topic：（慎用，只会删除zookeeper中的元数据，消息文件须手动删除）
kafka-topics --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --delete --topic xman-baseline-reach-sms-incomenew-test1
#查看是否存在topic
Administrator in apps\windows-terminal\current took 8s
❯ kafka-topics --describe --topic xman-baseline-reach-sms-incomenew-test1 --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092
Error while executing topic command : Topic 'xman-baseline-reach-sms-incomenew-test1' does not exist as expected
[2022-05-23 17:41:43,070] ERROR java.lang.IllegalArgumentException: Topic 'xman-baseline-reach-sms-incomenew-test1' does not exist as expected
at kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:542)
at kafka.admin.TopicCommand$AdminClientTopicService.describeTopic(TopicCommand.scala:317)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:69)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
(kafka.admin.TopicCommand$)
对分区进行扩容
kafka-topics --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --alter --topic xman-baseline-reach-sms-incomenew-test1 --partitions 4
Administrator in apps\windows-terminal\current took 7s
❯ kafka-topics --describe --topic xman-baseline-reach-sms-incomenew-test1 --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092
Topic: xman-baseline-reach-sms-incomenew-test1 PartitionCount: 4 ReplicationFactor: 2 Configs: min.insync.replicas=2,message.format.version=2.2-IV1,unclean.leader.election.enable=true
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 0 Leader: 1 Replicas: 1,3 Isr: 1,3
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 1 Leader: 3 Replicas: 3,2 Isr: 3,2
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 2 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 3 Leader: 1 Replicas: 1,2 Isr: 1,2
#只能扩容，不能缩小
Administrator in apps\windows-terminal\current
❯ kafka-topics --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --alter --topic xman-baseline-reach-sms-incomenew-test1 --partitions 3
Error while executing topic command : Topic currently has 4 partitions, which is higher than the requested 3.
[2022-05-23 17:54:24,754] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: Topic currently has 4 partitions, which is higher than the requested 3.
(kafka.admin.TopicCommand$)
生产消费
消费：
kafka-console-consumer --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --topic test-topic --from-beginning
hongda is a good man
whoami
{"name":"hongda","age":27,"city":"shanghai"}
生产：（跟上面是一个kafka集群）
kafka-console-producer --bootstrap-server b-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --topic test-topic
>hongda is a good man
>whoami
>{"name":"hongda","age":27,"city":"shanghai"}
>
消费者组：
kafka-consumer-groups --bootstrap-server b-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --list |grep cms-api-kafka
gid-xman-baseline-cms-api-kafka-20211220185102
gid-xman-baseline-cms-api-kafka-genesis-dev
查看消费组具体详情
kafka-consumer-groups --bootstrap-server b-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --describe --group gid-xman-baseline-cms-api-kafka-20211220185102
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
HOST CLIENT-ID
gid-xman-baseline-cms-api-kafka-20211220185102 xman-baseline-cms-api-kafka-xman-genesis-sit 0 9 9 0 RequestDetail-df254cfc-eea5-49a5-b065-7c9bda7141b6 /172.29.155.135 RequestDetail
gid-xman-baseline-cms-api-kafka-20211220185102 xman-baseline-cms-api-kafka-xman-genesis-sit 1 12 12 0 RequestDetail-df254cfc-eea5-49a5-b065-7c9bda7141b6 /172.29.155.135 RequestDetail
可以看到消费者组对应的topic
修改消费组的位偏移量
kafka-consumer-groups --bootstrap-server b-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --group gid-xman-baseline-cms-api-kafka-20211220185102 --reset-offsets --all-topics --by-duration PT0H30M0S
WARN: No action will be performed as the --execute option is missing.In a future major release, the default behavior of this command will be to prompt the user before executing the reset rather than doing a dry run. You should add the --dry-run option explicitly if you are scripting this command and want to keep the current default behavior without prompting.
Error: Assignments can only be reset if the group 'gid-xman-baseline-cms-api-kafka-20211220185102' is inactive, but the current state is Stable.
GROUP TOPIC PARTITION NEW-OFFSET
将消费组offset设置到最后30分钟
kafka-consumer-groups --bootstrap-server b-1.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --group gid-xman-baseline-cms-api-kafka-20211220185102 --reset-offsets --all-topics --to-latest --execute
Error: Assignments can only be reset if the group 'gid-xman-baseline-cms-api-kafka-20211220185102' is inactive, but the current state is Stable.
GROUP TOPIC PARTITION NEW-OFFSET
将消费组offset设置到最新
案例：
#创建topic
kafka-topics --create --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092 --replication-factor 2 --partitions 3 --topic xman-baseline-reach-sms-incomenew-test1
Created topic xman-baseline-reach-sms-incomenew-test1.
#查看topic详细信息
Administrator in apps\windows-terminal\current took 5s
❯ kafka-topics --describe --topic xman-baseline-reach-sms-incomenew-test1 --bootstrap-server b-3.awschina-nx-ms.njr84r.c4.kafka.cn-northwest-1.amazonaws.com.cn:9092
Topic: xman-baseline-reach-sms-incomenew-test1 PartitionCount: 3 ReplicationFactor: 2 Configs: min.insync.replicas=2,message.format.version=2.2-IV1,unclean.leader.election.enable=true
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 0 Leader: 1 Replicas: 1,3 Isr: 1,3
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 1 Leader: 3 Replicas: 3,2 Isr: 3,2
Topic: xman-baseline-reach-sms-incomenew-test1 Partition: 2 Leader: 2 Replicas: 2,1 Isr: 2,1
参考：
kafka_2.12-2.2.0在windows下运行（使用kafka内置zookeeper）
