shell实战之Linux主机系统监控
1、系统监控概述
采集的监控信息主要有内存占用率，CPU占用率，当前在线用户，磁盘挂载及磁盘空间使用率，平均每秒写入流量，平均每秒流出流量。磁盘IO：平均每秒从磁盘读入内存的速率，平均每秒从内存写入磁盘的速率。
2、监控原理
2.1、CPU占用率
监控原理：
CPU相关信息记录在文件 /proc/stat中。详情请查看博文：https://blog.csdn.net/ustclu/article/details/1721673
stephen@stephen-K55VD:~/shell$ cat /proc/stat cpu 348229 906 98356 7304276 81726 0 2821 0 0 0 cpu0 95033 273 22980 1803962 33023 0 1721 0 0 0 cpu1 79735 255 24756 1836717 17035 0 454 0 0 0 cpu2 84045 211 25742 1831963 16753 0 582 0 0 0 cpu3 89415 166 24876 1831633 14913 0 62 0 0 0 intr 10306028 7 28486 0 0 0 0 0 0 1 825 0 0 50130 0 0 0 76 284421 0 213811 0 0 0 29 795993 19 0 81 766580 15 648 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ctxt 51268973 btime 1554444493 processes 14526 procs_running 1 procs_blocked 0 softirq 9059312 7 2712077 5 5478 204089 0 1245879 2780432 0 2111345
代码实现：
1 #获取CPU的总量与使用量 2 cpuTotalStart=`awk 'BEGIN{total=0} /cpu / {for(i=2;i<=NF;i++);total+=i}END{print $total}' /proc/stat` 3 cpuUsedStart=`awk 'BEGIN{used=0} /cpu / { used=$2+$3+$4+$7+$8 }END{print used}' /proc/stat` 4 #隔30s再获取一次CPU总量与使用量并计算差值 5 sleep 30 6 cpuTotalEnd=`awk 'BEGIN{total=0} /cpu / {for(i=2;i<=NF;i++);total+=i}END{print $total}' /proc/stat` 7 cpuUsedEnd=`awk 'BEGIN{used=0} /cpu / { used=$2+$3+$4+$7+$8 }END{print used}' /proc/stat` 8 usedCPU=`expr ${cpuUsedEnd} - ${cpuUsedStart}` 9 totalCPU=`expr ${cpuTotalEnd} - ${cpuTotalStart}`
2.2、内存占用率
监控原理：
内存相关的信息记录在/proc/meminfo文件中，MemTotal为内存总量，单位为kb，MemFree为空闲内存。内存占用率=（总内存-空闲内存）/ 总内存。
stephen@stephen-K55VD:~/shell$ cat /proc/meminfo MemTotal: 3922884 kB MemFree: 139108 kB MemAvailable: 317700 kB Buffers: 31792 kB Cached: 538160 kB SwapCached: 10012 kB Active: 2615652 kB
代码实现：
1 #获取内存使用率 2 function memUsage(){ 3 logInfo "Begin to get mem usage of Host [${ip}]" 4 #获取总内存 5 totalMem=`awk '/MemTotal/{print $2}' /proc/meminfo` 6 #获取空闲内存 7 freeMem=`awk '/MemFree/{print $2}' /proc/meminfo` 8 usedMem=`expr ${totalMem} - ${freeMem}` 9 #echo $(usagePercent ${usedMem} ${totalMem}) 10 #echo $(kbToGb ${totalMem}) 11 logInfo "Host [${ip}] total mem is : $(kbToGb ${totalMem}) GB" 12 #计算内存使用率并打印到日志中 13 logInfo "Host [${ip}] mem usage is : $(usagePercent ${usedMem} ${totalMem}) %" 14 logInfo "End to get mem usage of Host [${ip}]" 15 }
2.3、流量监控
监控原理：
Linux机器流量信息记录在/proc/net/dev文件中。通过计算一段时间段内接收和发送的字节数来计算速率。第一列为网卡信息，第二列为接收的字节数，第10列为发送的字节数。
stephen@stephen-K55VD:~/shell/sysMonitor$ cat /proc/net/dev Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed wlp3s0: 19595253 41163 0 0 0 0 0 0 34741446 49185 0 0 0 0 0 0 enp4s0f2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 docker0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 lo: 907275 5032 0 0 0 0 0 0 907275 5032 0 0 0 0 0 0
代码实现：
1 #ethName为网卡名称 2 receiveByteStart=`cat /proc/net/dev |grep -E "${ethName}"|awk '{print $2}'` 3 sendByteStart=`cat /proc/net/dev |grep -E "${ethName}"|awk '{print $10}'`
2.4、磁盘IO
监控原理：
磁盘IO相关的信息记录在/proc/vmstat文件中，pgpgin对应的为输入方向的数据量。pgpgout对应的为输出方向的数据量。采集一段时间的数据量，除以时间来计算速率。
代码实现：
1 #disk IO in 2 function diskIOIn(){ 3 #获取磁盘入方向IO 4 inIoStart=`awk '/pgpgin/{print $2}' /proc/vmstat` 5 sleep 30 6 inIoEnd=`awk '/pgpgin/{print $2}' /proc/vmstat` 7 inIo=$(((inIoEnd-inIoStart)/(30*1024))) 8 logInfo "Host [${ip}] in IO is : ${inIo} MB / s" 9 10 }
3、脚本代码
- hostLists:监控主机的IP集合。
- sysMonitor.sh*：获取各项监控信息的脚本。
1 #!/bin/bash 2 #监控linux主机系统信息 3 #导入工具模块 4 source utils 5 6 #获取CPU占用率 7 function cpuUsage() 8 { 9 #物理CPU个数 10 phyCPUNums=`cat /proc/cpuinfo |grep "physical id"|sort |uniq|wc -l` 11 #逻辑CPU个数 12 lgCPUNums=`cat /proc/cpuinfo |grep "processor"|wc -l` 13 #core 14 cores=`cat /proc/cpuinfo |grep "cores"|uniq|awk '{print $4}'` 15 logInfo "Host [${ip}] physical CPU nums is : ${phyCPUNums}" 16 logInfo "Host [${ip}] logic CPU nums is : ${lgCPUNums}" 17 logInfo "Host [${ip}] core nums is : ${cores}" 18 #CPU占用率 19 #获取CPU的总量与使用量 20 cpuTotalStart=`awk 'BEGIN{total=0} /cpu / {for(i=2;i<=NF;i++);total+=i}END{print $total}' /proc/stat` 21 cpuUsedStart=`awk 'BEGIN{used=0} /cpu / { used=$2+$3+$4+$7+$8 }END{print used}' /proc/stat` 22 #隔30s再获取一次CPU总量与使用量并计算差值 23 sleep 30 24 cpuTotalEnd=`awk 'BEGIN{total=0} /cpu / {for(i=2;i<=NF;i++);total+=i}END{print $total}' /proc/stat` 25 cpuUsedEnd=`awk 'BEGIN{used=0} /cpu / { used=$2+$3+$4+$7+$8 }END{print used}' /proc/stat` 26 usedCPU=`expr ${cpuUsedEnd} - ${cpuUsedStart}` 27 totalCPU=`expr ${cpuTotalEnd} - ${cpuTotalStart}` 28 logInfo "Host [${ip}] CPU usage is : $(usagePercent ${usedCPU} ${totalCPU}) %" 29 30 } 31 32 #获取内存使用率 33 function memUsage(){ 34 logInfo "Begin to get mem usage of Host [${ip}]" 35 #获取总内存 36 totalMem=`awk '/MemTotal/{print $2}' /proc/meminfo` 37 #获取空闲内存 38 freeMem=`awk '/MemFree/{print $2}' /proc/meminfo` 39 usedMem=`expr ${totalMem} - ${freeMem}` 40 #echo $(usagePercent ${usedMem} ${totalMem}) 41 #echo $(kbToGb ${totalMem}) 42 logInfo "Host [${ip}] total mem is : $(kbToGb ${totalMem}) GB" 43 #计算内存使用率并打印到日志中 44 logInfo "Host [${ip}] mem usage is : $(usagePercent ${usedMem} ${totalMem}) %" 45 logInfo "End to get mem usage of Host [${ip}]" 46 } 47 48 #网卡平均每秒流量 49 function netData(){ 50 logInfo "Begin to get net data of Host [${ip}]" 51 ethName=$1 52 receiveByteStart=`cat /proc/net/dev |grep -E "${ethName}"|awk '{print $2}'` 53 sendByteStart=`cat /proc/net/dev |grep -E "${ethName}"|awk '{print $10}'` 54 sleep 10 55 receiveByteSEnd=`cat /proc/net/dev |grep -E "${ethName}"|awk '{print $2}'` 56 sendBytesEnd=`cat /proc/net/dev |grep -E "${ethName}"|awk '{print $10}'` 57 inDataRate=$(echo "scale=2;(${receiveByteSEnd}-${receiveByteStart})/10" | bc) 58 outDataRate=$(echo "scale=2;(${sendBytesEnd}-${sendByteStart})/10" | bc) 59 logInfo "Host [${ip}] in data is : ${inDataRate} kb / s" 60 logInfo "Host [${ip}] out data is : ${outDataRate} kb / s" 61 logInfo "End to get net data of Host [${ip}]" 62 } 63 64 #磁盘空间使用情况 65 function diskUsage(){ 66 logInfo "Begin to get disk usage of Host [${ip}]" 67 noTimeLogInfo "`df -h`" 68 logInfo "End to get disk usage of Host [${ip}]" 69 } 70 71 #disk IO in 72 function diskIOIn(){ 73 #获取磁盘入方向IO 74 inIoStart=`awk '/pgpgin/{print $2}' /proc/vmstat` 75 sleep 30 76 inIoEnd=`awk '/pgpgin/{print $2}' /proc/vmstat` 77 inIo=$(((inIoEnd-inIoStart)/(30*1024))) 78 logInfo "Host [${ip}] in IO is : ${inIo} MB / s" 79 80 } 81 82 #disk IO out 83 function diskIOout(){ 84 #获取磁盘出方向的IO 85 outIoStart=`awk '/pgpgout/{print $2}' /proc/vmstat` 86 sleep 60 87 outIoEnd=`awk '/pgpgout/{print $2}' /proc/vmstat` 88 outIo=$(((outIoEnd-outIoStart)/(60*1024))) 89 logInfo "Host [${ip}] out IO is : ${outIo} MB / s" 90 } 91 92 #当前在线用户 93 function onlineUser(){ 94 user=`w |awk 'NR>1'|awk '{print $1 "\t" "\t" $4}'` 95 userCount=`w |awk 'NR>1'|wc -l` 96 #loginAt=`w |awk 'NR>1'|awk '{print $4 }'` 97 logInfo "There are [${userCount}] users online now." 98 noTimeLogInfo "UserName loginAt" 99 noTimeLogInfo "${user}" 100 } 101 102 #判断主机网络连通性 103 function isAlive(){ 104 for ip in `cat hostLists` 105 do 106 ping ${ip} -c 3 >/dev/null 107 if [ $? -eq 0 ];then 108 logInfo "${ip} is reachable" 109 #查看在线用户 110 onlineUser 111 #获取CPU相关信息 112 cpuUsage 113 #获取mem相关信息 114 memUsage 115 #获取磁盘IO 116 diskIOIn 117 diskIOout 118 #磁盘使用率 119 diskUsage 120 #平均每秒流接收或输出流量 121 netData wlp3s0 122 else 123 logInfo "ERROR ${ip} is unreachable,try login in see more details.." 124 fi 125 done 126 } 127 128 while [ 1 ] 129 do 130 isAlive 131 sleep 60 132 done
- utils：打印日志的函数等。
1 #!/bin/bash 2 #日志打印 3 curr_path=`pwd` 4 function logInfo() 5 { 6 local curr_time=`date "+%Y-%m-%d %H:%M:%S"` 7 log_file=${curr_path}/system_status.log 8 #判断日志文件是否存在 9 if [ -e ${log_file} ] 10 then 11 #检测文件是否可写 12 if [ -w ${log_file} ] 13 then 14 #若文件无写权限则使用chmod命令赋予权限 15 chmod 770 ${log_file} 16 fi 17 else 18 #若日志文件不存在则创建 19 touch ${log_file} 20 fi 21 #写日志 22 local info=$1 23 echo "${curr_time} `whoami` [Info] ${info}">>${log_file} 24 } 25 function noTimeLogInfo(){ 26 msg=$1 27 echo "${msg}">>${log_file} 28 } 29 30 #把kb转换成gb，精度为3。expr只支持整数计算 31 function kbToGb(){ 32 kbVal=$1 33 gbVal=$(echo "scale=3;${kbVal}/1024/1024"| bc) 34 echo $gbVal 35 } 36 #使用率以百分比的形式 37 #第一个参数为已使用量，第二个参数为总量 38 function usagePercent(){ 39 used=$1 40 total=$2 41 usedPercent=$(echo "scale=2;${used}*100/${total}"| bc) 42 echo ${usedPercent} 43 }
脚本结构：
1 -rw-r--r-- 1 stephen stephen 30 4月 5 18:33 hostLists 2 -rwxrwxr-x 1 stephen stephen 4164 4月 5 18:50 sysMonitor.sh* 3 -rw-r--r-- 1 stephen stephen 951 4月 5 15:23 utils
4、运行结果
监控信息记录在日志system_status.log中。运行结果如下：
2019-04-05 19:44:42 stephen [Info] 192.168.1.109 is reachable 2019-04-05 19:44:42 stephen [Info] There are [2] users online now. UserName loginAt USER LOGIN@ stephen 14:09 2019-04-05 19:44:42 stephen [Info] Host [192.168.1.109] physical CPU nums is : 1 2019-04-05 19:44:42 stephen [Info] Host [192.168.1.109] logic CPU nums is : 4 2019-04-05 19:44:42 stephen [Info] Host [192.168.1.109] core nums is : 2 2019-04-05 19:45:12 stephen [Info] Host [192.168.1.109] CPU usage is : 10.12 % 2019-04-05 19:45:12 stephen [Info] Begin to get mem usage of Host [192.168.1.109] 2019-04-05 19:45:12 stephen [Info] Host [192.168.1.109] total mem is : 3.741 GB 2019-04-05 19:45:12 stephen [Info] Host [192.168.1.109] mem usage is : 95.83 % 2019-04-05 19:45:12 stephen [Info] End to get mem usage of Host [192.168.1.109] 2019-04-05 19:45:42 stephen [Info] Host [192.168.1.109] in IO is : 0 MB / s 2019-04-05 19:46:42 stephen [Info] Host [192.168.1.109] out IO is : 0 MB / s 2019-04-05 19:46:42 stephen [Info] Begin to get disk usage of Host [192.168.1.109] 文件系统 容量 已用 可用 已用% 挂载点 udev 1.9G 0 1.9G 0% /dev tmpfs 384M 2.0M 382M 1% /run /dev/sda10 42G 20G 20G 51% / tmpfs 1.9G 20M 1.9G 2% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/loop0 3.8M 3.8M 0 100% /snap/notepad-plus-plus/202 /dev/loop2 54M 54M 0 100% /snap/core18/782 /dev/loop4 441M 441M 0 100% /snap/wine-platform/111 /dev/loop5 441M 441M 0 100% /snap/wine-platform/105 /dev/loop7 3.8M 3.8M 0 100% /snap/notepad-plus-plus/199 /dev/loop3 90M 90M 0 100% /snap/core/6673 /dev/loop1 274M 274M 0 100% /snap/wps-office-multilang/1 /dev/loop6 91M 91M 0 100% /snap/core/6405 /dev/loop8 92M 92M 0 100% /snap/core/6531 /dev/loop9 36M 36M 0 100% /snap/gtk-common-themes/1198 /dev/loop10 3.8M 3.8M 0 100% /snap/notepad-plus-plus/195 /dev/loop11 441M 441M 0 100% /snap/wine-platform/103 tmpfs 384M 16K 384M 1% /run/user/125 tmpfs 384M 52K 384M 1% /run/user/1000 2019-04-05 19:46:42 stephen [Info] End to get disk usage of Host [192.168.1.109] 2019-04-05 19:46:42 stephen [Info] Begin to get net data of Host [192.168.1.109] 2019-04-05 19:46:52 stephen [Info] Host [192.168.1.109] in data is : 42.90 kb / s 2019-04-05 19:46:52 stephen [Info] Host [192.168.1.109] out data is : 7.00 kb / s 2019-04-05 19:46:52 stephen [Info] End to get net data of Host [192.168.1.109] 2019-04-05 19:47:04 stephen [Info] ERROR 255.255.255.254 is unreachable,try login in see more details..
5、参考文档
5.1、ifstat网络流量监控之/proc/net/dev文件
https://blog.csdn.net/kongshuai19900505/article/details/80676607
5.2、awk命令
http://man.linuxde.net/awk
5.3、使用shell脚本采集系统cpu、内存、磁盘、网络等信息
https://www.jb51.net/article/50436.htm
