【Azure 事件中心】在Windows系统中使用 kafka-consumer-groups.bat 查看Event Hub中kafka的consumer groups信息

问题描述

 

使用前提

  1. 在Azure中创建一个支持Kafka的Event Hub(标准层及以上)
  2. 下载示例代码:git clone https://github.com/Azure/azure-event-hubs-for-kafka.git 
  3. 修改 producer.config 和  consumer.config

 

操作步骤

If customer has an active group, they can check consumer lag (which indicate what data has been consumed) through Kafka's own kafka-consumer-groups.sh command (https://kafka.apache.org/documentation/#basic_ops_consumer_lag)

  • Make sure you setup Java JDK on your machine. then download the Kafka binary downloads from (https://kafka.apache.org/downloads) - you just need the binary downloads, no need to get the source unless you intend to modify the source code that the script uses.
  • from command prompt, navigate to your downloaded bin directory - e.g. F:\kafka_2.13-3.0.0\bin\windows (for windows shell script).
  • Create a properties file that has the connection string for your namespace. For example you can create a config.properties file that has the following.
bootstrap.servers=<your namespace>.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="<Your connection string>";

 

  • After that just run through the command as describe in https://kafka.apache.org/documentation/#basic_ops_consumer_lag to see your information. sample below. Note that currently, this only works if the group is an active group meaning it has active consumers running for the group. this command does not work for EventHub service if the group is inactive.
F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server <your namespace>.servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --list
$hmlam-4

F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server <your namespace>.servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --describe --group $hmlam-4

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                        HOST            CLIENT-ID
$hmlam-4        hmlam-eh-2      1          52648           52651           3               hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-4:I:perf_client#1-c391de052f2446ff85fbf6899160a236 0.0.0.0         perf_client#1
$hmlam-4        hmlam-eh-2      0          52932           52946           14              hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-

 

目前遇见的问题

当根据以上步骤设置好,并且同时运行了生产者代码和消费组代码,可是却无法获取到kafka group信息

 

(TODO: 问题还在进一步的研究中)

 

posted @   路边两盏灯  阅读(235)  评论(0编辑  收藏  举报
编辑推荐:
· Linux系列:如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列(二):开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 探究高空视频全景AR技术的实现原理
· 理解Rust引用及其生命周期标识(上)
阅读排行:
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布:重大改进与新特性概览!
· AI与.NET技术实操系列(二):开始使用ML.NET
· 单线程的Redis速度为什么快?
历史上的今天:
2020-11-18 【Azure Redis 缓存 Azure Cache For Redis】Redis支持的版本及不同版本迁移风险
点击右上角即可分享
微信分享提示