Fork me on GitHub

Kafka监控| Manager| Monitor| Eagle

 

1.kafka Manager

1.上传压缩包kafka-manager-1.3.3.15.zip到集群

2.解压到/opt/module

3.修改配置文件conf/application.conf
kafka-manager.zkhosts="kafka-manager-zookeeper:2181"
修改为:
kafka-manager.zkhosts="hadoop101:2181,hadoop102:2181,hadoop103:2181"

4.启动kafka-manager
bin/kafka-manager

5.登录hadoop102:9000页面查看详细信息
[kris@hadoop101 software]$ unzip kafka-manager-1.3.3.15.zip -d /opt/module/
[kris@hadoop101 bin]$ chmod +x kafka-manager
[kris@hadoop101 bin]$ ./kafka-manager ##启动

9000端口号可能会冲突,也可以修改端口号

启动KafkaManager
[kris@hadoop101 kafka-manager]$ 
nohup bin/kafka-manager   -Dhttp.port=7456 >/opt/module/kafka-manager-1.3.3.22/start.log 2>&1 &

 

添加集群:

 集群信息:

添加主题;

重新获取平衡

 

分区数、节点数、Brokers Spread节点分布比例(如一共3个节点,主题占了3个节点所以分布比例是100%,如果有3个节点主题只占1个,则33% )

Brokers Skew节点倾斜度、(1个Topic--3个分区--每个分区3个副本;如果某个节点上分区数>3个, 有节点超负荷了(有1个节点超负荷了即1/3=33%,两个节点超负荷了则2/3)

Brokers Leader Skew Leader的倾斜度,leader都在一个节点上,leader倾斜度)

 

 起一个消费者

[kris@hadoop101 kafka]$ bin/kafka-console-consumer.sh --bootstrap-server hadoop101:9092 --topic first

 

2. KafkaMonitor

局限性,offset在zk上可以使用

1.上传jar包KafkaOffsetMonitor-assembly-0.2.0.jar到集群

2.在/opt/module/下创建kafka-offset-console文件夹

3.将上传的jar包放入刚创建的目录下

4.在/opt/module/kafka-offset-console目录下创建启动脚本start_en.sh,内容如下:
java -cp就是执行下那个jar包;主方法所在的类的类名;zk地址;每隔10s刷新下库; [kris@hadoop101 kafka
-offset-console]$ vim start_en.sh #!/bin/bash java -Xms512M -Xmx512M -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=512m -cp KafkaOffsetMonitor-assembly-0.2.0.jar \ com.quantifind.kafka.offsetapp.OffsetGetterWeb \ --zk 192.168.1.101:2181,192.168.1.102:2182,192.168.1.103:2181 \ --port 8086 \ --refresh 10.seconds \ --retain 7.days 1>mobile-logs/stdout.log 2>mobile-logs/stderr.log & [kris@hadoop101 kafka-offset-console]$ ll 总用量 56560 -rw-rw-r--. 1 kris kris 57910726 3月 1 21:23 KafkaOffsetMonitor-assembly-0.2.0.jar -rw-rw-r--. 1 kris kris 375 3月 1 21:25 start_en.sh 5.在/opt/module/kafka-offset-console目录下创建mobile-logs文件夹 [kris@hadoop101 kafka-offset-console]$ mkdir /opt/module/kafka-offset-console/mobile-logs [kris@hadoop101 kafka-offset-console]$ ll 总用量 56564 -rw-rw-r--. 1 kris kris 57910726 3月 1 21:23 KafkaOffsetMonitor-assembly-0.2.0.jar drwxrwxr-x. 2 kris kris 4096 3月 1 21:26 mobile-logs -rw-rw-r--. 1 kris kris 375 3月 1 21:25 start_en.sh [kris@hadoop101 kafka-offset-console]$ ./start_en.sh -bash: ./start_en.sh: 权限不够 [kris@hadoop101 kafka-offset-console]$ chmod +x start_en.sh 6.启动KafkaMonitor [kris@hadoop101 kafka-offset-console]$ ./
start_en.sh

 http://hadoop101:8086/#/

 

[kris@hadoop101 kafka]$ bin/kafka-console-producer.sh --broker-list hadoop101:9092 --topic first
>hello
>world
[kris@hadoop101 kafka]$ bin/kafka-console-consumer.sh --zookeeper hadoop101:2181 --topic first                    
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
hello
world
java

 

主题生产到了longSize184的offset,目前消费到177,lag还有6个没有消费到;

 lag有多少条消息没有被消费;到最后都被消费掉lag=0

 

3. Kafka Eagle

1)修改kafka启动命令

修改kafka-server-start.sh命令中

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fiif [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
    #export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi
注意:修改之后在启动Kafka之前要分发之其他节点

2)上传压缩包kafka-eagle-bin-1.3.7.tar.gz到集群/opt/software目录

3)解压到本地

[kris@hadoop101 software]$ tar -zxvf kafka-eagle-bin-1.3.7.tar.gz

4)进入刚才解压的目录

[kris@hadoop101 kafka-eagle-bin-1.3.7]$ ll

-rw-rw-r--. 1 kris kris 84920710 8月  13 23:00 kafka-eagle-web-1.3.7-bin.tar.gz

5)将kafka-eagle-web-1.3.7-bin.tar.gz解压至/opt/module

[kris@hadoop101 kafka-eagle-bin-1.3.7]$ tar -zxvf kafka-eagle-web-1.3.7-bin.tar.gz -C /opt/module/

6)修改名称

[kris@hadoop101 module]$ mv kafka-eagle-web-1.3.7/ eagle

7)给启动文件执行权限

[kris@hadoop101 bin]$ chmod 777 ke.sh

8)修改配置文件

######################################

# multi zookeeper&kafka cluster list

######################################

kafka.eagle.zk.cluster.alias=cluster1

cluster1.zk.list=hadoop101:2181,hadoop102:2181,hadoop103:2181

 

######################################

# kafka offset storage

######################################

cluster1.kafka.eagle.offset.storage=kafka

 

######################################

# enable kafka metrics

######################################

kafka.eagle.metrics.charts=true

kafka.eagle.sql.fix.error=false

 

######################################

# kafka jdbc driver address

######################################

kafka.eagle.driver=com.mysql.jdbc.Driver

kafka.eagle.url=jdbc:mysql://hadoop101:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull

kafka.eagle.username=root

kafka.eagle.password=123456

 

9)添加环境变量

export KE_HOME=/opt/module/eagle

export PATH=$PATH:$KE_HOME/bin

注意:source /etc/profile

10)启动

[kris@hadoop101 eagle]$ bin/ke.sh start

... ...

*******************************************************************

* Kafka Eagle Service has started success.

* Welcome, Now you can visit 'http://192.168.9.101:8048/ke'

* Account:admin ,Password:123456

*******************************************************************

* <Usage> ke.sh [start|status|stop|restart|stats] </Usage>

* <Usage> https://www.kafka-eagle.org/ </Usage>

*******************************************************************

注意:启动之前需要先启动ZK以及KAFKA

11)登录页面查看监控数据

http://192.168.9.101:8048/ke

 

posted @ 2019-03-03 00:43  kris12  阅读(1629)  评论(0编辑  收藏  举报
levels of contents