kafka可视化web工具kafdrop

一、Kafdrop是Apache Kafka的开源Web UI可视化界面 

主要功能

1、查看Kafka代理 -主题和分区分配以及控制器状态

2、查看主题 -分区数，复制状态和自定义配置

3、浏览消息 -JSON，纯文本和Avro编码

4、查看消费者组 -每个分区的停放偏移量，合并延迟和每个分区滞后

5、创建新主题

6、查看ACL

二、使用docker-compose安装kafdrop

编写docker-compose-kafdrop.yaml文件

[root@redis-01 kafdrop]# mkdir -p /home/kafdrop
[root@redis-01 kafdrop]# vim /home/kafdrop/docker-compose-kafdrop.yaml

version: "3"
services:
  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka120.blockchain.hl95.com:9092"

三、安装并启动kafdrop

[root@redis-01 kafdrop]# docker-compose -f docker-compose-kafdrop.yaml up -d
Creating network "kafdrop_default" with the default driver
Pulling kafdrop (obsidiandynamics/kafdrop:)...
latest: Pulling from obsidiandynamics/kafdrop
ad18637d63f5: Pull complete
ef2c93782772: Pull complete
7d4c0bffca0a: Pull complete
13a3e19748f8: Pull complete
49caf6a52b53: Pull complete
d1a2c2d153cb: Pull complete
Digest: sha256:b7ba8577ce395b1975b0ed98bb53cb6b13e7d32d5442188da1ce41c0838d1ce9
Status: Downloaded newer image for obsidiandynamics/kafdrop:latest
Creating kafdrop_kafdrop_1 ... done

第一次运行docker-compose会先下载obsidiandynamics/kafdrop镜像，随后启动容器。

查看容器运行状态

[root@redis-01 kafdrop]# docker ps -a
CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS                      PORTS               NAMES
6ccf05249e55        obsidiandynamics/kafdrop   "/kafdrop.sh"       25 seconds ago      Exited (1) 17 seconds ago                       kafdrop_kafdrop_1
[root@redis-01 kafdrop]# 

发现容器自动退出，查看容器日志

[root@redis-01 kafdrop]# docker logs kafdrop_kafdrop_1

:::    :::     :::     :::::::::: :::::::::  :::::::::   ::::::::  :::::::::        ::::::::  
:+:   :+:    :+: :+:   :+:        :+:    :+: :+:    :+: :+:    :+: :+:    :+:      :+:    :+: 
+:+  +:+    +:+   +:+  +:+        +:+    +:+ +:+    +:+ +:+    +:+ +:+    +:+             +:+ 
+#++:++    +#++:++#++: :#::+::#   +#+    +:+ +#++:++#:  +#+    +:+ +#++:++#+           +#++:  
+#+  +#+   +#+     +#+ +#+        +#+    +#+ +#+    +#+ +#+    +#+ +#+                    +#+ 
#+#   #+#  #+#     #+# #+#        #+#    #+# #+#    #+# #+#    #+# #+#             #+#    #+# 
###    ### ###     ### ###        #########  ###    ###  ########  ###              ########  

2020-07-02 01:45:18.990  INFO ${sys:PID} [           main] k.Kafdrop$EnvironmentSetupListener       : Initializing JAAS config
2020-07-02 01:45:19.013  INFO ${sys:PID} [           main] k.Kafdrop$EnvironmentSetupListener       : env: null .isSecured kafka: false
2020-07-02 01:45:19.015  INFO ${sys:PID} [           main] k.Kafdrop$EnvironmentSetupListener       : Env: null
2020-07-02 01:45:19.290  INFO 1 [           main] o.s.b.StartupInfoLogger                  : Starting application on 6ccf05249e55 with PID 1 (started by root in /)
2020-07-02 01:45:19.292  INFO 1 [           main] o.s.b.SpringApplication                  : No active profile set, falling back to default profiles: default
2020-07-02 01:45:22.254  INFO 1 [           main] i.u.s.s.ServletContextImpl               : Initializing Spring embedded WebApplicationContext
2020-07-02 01:45:22.255  INFO 1 [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2877 ms
2020-07-02 01:45:23.012  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking truststore file kafka.truststore.jks
2020-07-02 01:45:23.013  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking keystore file kafka.keystore.jks
2020-07-02 01:45:23.013  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking properties file kafka.properties
2020-07-02 01:45:23.210  WARN 1 [           main] o.a.k.c.ClientUtils                      : Couldn't resolve server kafka120.blockchain.hl95.com:9092 from bootstrap.servers as DNS resolution failed for kafka120.blockchain.hl95.com
2020-07-02 01:45:23.215  WARN 1 [           main] o.s.c.s.AbstractApplicationContext       : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'aclController' defined in URL [jar:file:/kafdrop-3.27.0/kafdrop-3.27.0.jar!/BOOT-INF/classes!/kafdrop/controller/AclController.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaMonitorImpl' defined in URL [jar:file:/kafdrop-3.27.0/kafdrop-3.27.0.jar!/BOOT-INF/classes!/kafdrop/service/KafkaMonitorImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaHighLevelConsumer': Invocation of init method failed; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
2020-07-02 01:45:23.241  INFO 1 [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-07-02 01:45:23.244 ERROR 1 [           main] o.s.b.SpringApplication                  : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'aclController' defined in URL [jar:file:/kafdrop-3.27.0/kafdrop-3.27.0.jar!/BOOT-INF/classes!/kafdrop/controller/AclController.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaMonitorImpl' defined in URL [jar:file:/kafdrop-3.27.0/kafdrop-3.27.0.jar!/BOOT-INF/classes!/kafdrop/service/KafkaMonitorImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaHighLevelConsumer': Invocation of init method failed; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:769)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:218)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1341)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1187)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:845)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
        at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
        at kafdrop.Kafdrop.main(Kafdrop.java:53)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaMonitorImpl' defined in URL [jar:file:/kafdrop-3.27.0/kafdrop-3.27.0.jar!/BOOT-INF/classes!/kafdrop/service/KafkaMonitorImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaHighLevelConsumer': Invocation of init method failed; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:769)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:218)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1341)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1187)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1251)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1171)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760)
        ... 26 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaHighLevelConsumer': Invocation of init method failed; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:139)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1770)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1251)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1171)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760)
        ... 40 more
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:827)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)
        at kafdrop.service.KafkaHighLevelConsumer.initializeClient(KafkaHighLevelConsumer.java:48)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
        ... 53 more
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
        at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:88)
        at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:47)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:735)
        ... 63 more

[root@redis-01 kafdrop]# 

发现报错：2020-07-02 01:45:23.210 WARN 1 [ main] o.a.k.c.ClientUtils : Couldn't resolve server kafka120.blockchain.hl95.com:9092 from bootstrap.servers as DNS resolution failed for kafka120.blockchain.hl95.com

容器里不能解析宿主机配置的域名解析，需要在compose文件里使用extra_hosts指定域名配置

[root@redis-01 kafdrop]# vim docker-compose-kafdrop.yaml

最后添加：

extra_hosts:
  - "kafka120.blockchain.hl95.com:172.28.5.120"
  - "kafka124.blockchain.hl95.com:172.28.5.124"
  - "kafka125.blockchain.hl95.com:172.28.5.125"

删除之前的容器，重新运行

[root@redis-01 kafdrop]# docker-compose -f docker-compose-kafdrop.yaml up -d
Creating kafdrop_kafdrop_1 ... done
[root@redis-01 kafdrop]# docker ps -a
CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS              PORTS                    NAMES
ceda56d75e2e        obsidiandynamics/kafdrop   "/kafdrop.sh"       22 seconds ago      Up 21 seconds       0.0.0.0:9000->9000/tcp   kafdrop_kafdrop_1
[root@redis-01 kafdrop]# 

五、启动成功，访问ip:9000

 

 

 访问成功。