【Azure 事件中心】在Windows系统中使用 kafka-consumer-groups.bat 查看Event Hub中kafka的consumer groups信息
问题描述
使用 Apache Flink 连接支持 Apache Kafka的Azure Event Hub后,由于消费端的Consumer Group是动态创建,在门户页面和Service Bus Explorer工具中均无法查看,所以如何来查看 Kafka consumer group呢?可以参考以下内容:
使用前提
将 Apache Flink 与适用于 Apache Kafka 的 Azure 事件中心配合使用: https://docs.azure.cn/zh-cn/event-hubs/event-hubs-kafka-flink-tutorial
- 在Azure中创建一个支持Kafka的Event Hub(标准层及以上)
- 下载示例代码:git clone https://github.com/Azure/azure-event-hubs-for-kafka.git
- 修改 producer.config 和 consumer.config
操作步骤
If customer has an active group, they can check consumer lag (which indicate what data has been consumed) through Kafka's own kafka-consumer-groups.sh command (https://kafka.apache.org/documentation/#basic_ops_consumer_lag)
- Make sure you setup Java JDK on your machine. then download the Kafka binary downloads from (https://kafka.apache.org/downloads) - you just need the binary downloads, no need to get the source unless you intend to modify the source code that the script uses.
- from command prompt, navigate to your downloaded bin directory - e.g. F:\kafka_2.13-3.0.0\bin\windows (for windows shell script).
- Create a properties file that has the connection string for your namespace. For example you can create a config.properties file that has the following.
bootstrap.servers=<your namespace>.servicebus.windows.net:9093 security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="<Your connection string>";
- After that just run through the command as describe in https://kafka.apache.org/documentation/#basic_ops_consumer_lag to see your information. sample below. Note that currently, this only works if the group is an active group meaning it has active consumers running for the group. this command does not work for EventHub service if the group is inactive.
F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server <your namespace>.servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --list $hmlam-4 F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server <your namespace>.servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --describe --group $hmlam-4 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID $hmlam-4 hmlam-eh-2 1 52648 52651 3 hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-4:I:perf_client#1-c391de052f2446ff85fbf6899160a236 0.0.0.0 perf_client#1 $hmlam-4 hmlam-eh-2 0 52932 52946 14 hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-
目前遇见的问题
当根据以上步骤设置好,并且同时运行了生产者代码和消费组代码,可是却无法获取到kafka group信息
(TODO: 问题还在进一步的研究中)
当在复杂的环境中面临问题,格物之道需:浊而静之徐清,安以动之徐生。 云中,恰是如此!
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· Linux系列:如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列(二):开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 探究高空视频全景AR技术的实现原理
· 理解Rust引用及其生命周期标识(上)
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布:重大改进与新特性概览!
· AI与.NET技术实操系列(二):开始使用ML.NET
· 单线程的Redis速度为什么快?
2020-11-18 【Azure Redis 缓存 Azure Cache For Redis】Redis支持的版本及不同版本迁移风险