记录~采集机器出现load异常
症状描述
1、机器load增加从原来的不到1,疯狂增长到20+
2、问题期间,磁盘容量正常、cpu正常、内存正常
解决办法
根据load问题,查找磁盘读写,使用iotop命令查看哪些进程磁盘读写异常
iotop命令参数
-o:只显示有io操作的进程
-b:批量显示,无交互,主要用作记录到文件。
-n NUM:显示NUM次,主要用于非交互式模式。
-d SEC:间隔SEC秒显示一次。
-p PID:监控的进程pid。
-u USER:监控的进程用户。
快捷键
左右箭头:改变排序方式,默认是按IO排序。
r:改变排序顺序。
o:只显示有IO输出的进程。
p:进程/线程的显示方式的切换。
a:显示累积使用量。
q:退出。
查出flume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink进程读写磁盘异常如下,修改java配置调高内存,并重启后问题解决。
9572 be/4 flume 479.49 K/s 1302.39 K/s 0.00 % 6.21 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
9569 be/4 flume 664.65 K/s 2.25 M/s 0.00 % 6.10 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
9576 be/4 flume 749.31 K/s 2.08 M/s 0.00 % 3.96 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
9632 be/4 flume 0.00 B/s 407.49 K/s 0.00 % 3.84 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
9574 be/4 flume 357.64 K/s 1482.79 K/s 0.00 % 3.70 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
9566 be/4 flume 380.59 K/s 1899.78 K/s 0.00 % 2.97 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
9564 be/4 flume 531.72 K/s 1708.30 K/s 0.00 % 2.34 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/*::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-