Linux下monit进程管理操作梳理

 

Monit对运维人员来说可谓神器，它是一款功能非常丰富的进程、文件、目录和设备的监测工具，用于Unix平台。它可以自动修复那些已经停止运作的程序，特使适合处理那些由于多种原因导致的软件错误。
Monit不但本地监控十分有效，还可以监控远程服务，只要花点功夫就能永远实现服务的“死而复生”，就是说它可以使它监控的服务程序在宕停后迅速自启动，不需要人工干预。绝对牛X的一款系统监控神奇！
比如下面两个场景：
1）持续邮件提醒
默认情况下，如果服务Down了，无论它持续Down了多久，Monit程序只会邮件提醒你一次。下一次提醒，就是服务恢复的时候。
如果希望，在多个周期内，即使服务状态没有变化（持续宕机着），也能收到邮件提醒，那么加上这句：
alert foo@bar with reminder on 10 cycles 此句表示，在10个周期内都会邮件提醒。

2）误报提醒解决
有些时候，Monit也会误报，这很正常，任何监控软件都会。大多数是由于网络状况不佳。比如某一个服务，Monit发现停了，又迅速启动了，那就不要来烦了，别总是一封邮件接着一封。这样设置：
if failed host 172.16.5.1 port 8599 for 3 times within 4 cycles then alert  这样就是:若在四个周期内，三次 8599（我的电驴口）端口都无法通，则邮件通知。很方便！

废话不多说，下面对monit监控环境的部署做一梳理：

需求说明：
随着线上服务器数量的增加，各种开源软件和工具的广泛使用，一些服务自动停止或无响应的情况时有发生，其中有很大一部分是由于软件自身的稳定性或者机器硬件资源限制而造成的。按道理来讲，这些情况都应该设法找到本质原因，然后避免再次出现。但现实是残酷的，不少软件本身的稳定性有待提升，机器的硬件资源提升会触及成本，因此在集群的环境中，具备冗余，使得执行简单的服务重启成为了最现实的选择。这本身不是什么困难的事情，实现的方法有很多，比如在Zabbix或Nagios的报警中增加Action或Commands，或自己写脚本放到计划任务中执行都可以。然而下面要介绍的就是专门来做这种事情的一个工具：Monit。它最大的特点是配置文件简单易读，同时支持进程和系统状态的监控，并灵活的提供了各种检测的方式，周期，并进行报警和响应（重启服务，执行命令等）

Centos6下部署Monit环境过程：
1）安装EPEL仓库
[root@bastion-IDC src]# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
[root@bastion-IDC src]# rpm -ivh epel-release-latest-6.noarch.rpm

2）安装monit
[root@bastion-IDC src]# yum install -y monit

--------------------------------------------------------------------------------------------------------------
源码安装包下载：https://mmonit.com/monit/dist/binary/
# wget https://mmonit.com/monit/dist/binary/5.20.0/monit-5.20.0-linux-x86.tar.gz
# tar -zvxf monit-5.20.0-linux-x86.tar.gz
# mv monit-5.20.0 /usr/local/monit
# cp /usr/local/monit/conf/monitrc /etc/
然后编辑配置文件/etc/monitrc即可。centos6的配置文件是monit.conf，centos7的配置文件是monitrc
-------------------------------------------------------------------------------------------------------------

3）monit配置说明（官网配置说明：https://mmonit.com/monit/documentation/monit.html）
Monit配置文件/etc/monit.conf，可以将默认配置文件备份下，然后自定义配置
[root@bastion-IDC src]# cp /etc/monit.conf /etc/monit.conf.bak
[root@bastion-IDC src]# cat /etc/monit.conf         //自定义配置如下
set daemon 120              #Poll at 2-minute intervals         //每2分钟检查一次，单位为秒；monit做不到实时监控。
set logfile /home/monit/log/monit.log                  //monit的日志文件
set alert zhouwei@chinabank.com.cn with reminder on 1 cycle      //出现1次错误就发报警邮件到指定邮箱。多个邮箱地址就配置多行;with后的配置可以不加。
#set mailserver mail.tildeslash.com, mail.foo.bar port 10025, localhost with timeout 15 seconds
set mailserver 10.10.9.109             //设置邮件服务器
set httpd port 2812 and use address 10.10.8.2            //设置http监控页面的端口和ip
allow localhost           #Allow localhost to connect        //允许本机访问
allow 10.10.8.0/24            //允许此IP段访问
allow admin:nishiwode      #Allow Basic Auth          //认证的用户名和密码
# all system               //平均负载.内存使用率,cpu使用率
check system 10.10.8.2
   if loadavg (1min) > 4 then alert
   if loadavg (5min) > 2 then alert
   if memory usage > 75% then alert
   if cpu usage (user) > 70% then alert
   if cpu usage (system) > 30% then alert
   if cpu usage (wait) > 20% then alert
# all disk                    //磁盘空间使用率
check device data with path /dev/sda2
   if space usage > 90% then alert
   if inode usage > 85% then alert
check device home with path /dev/sda3
   if space usage > 85% for 5 cycles then alert      //如果在5个监控周期内，space使用率超过85%就发报警邮件  
   if inode usage > 85% for 5 cycles then alert
# all rsync
#10.10.8.2
check process sshd with pidfile /var/run/sshd.pid        //监控ssh服务
   start program "/etc/init.d/sshd start"
   stop program "/etc/init.d/sshd stop"
   if failed host 127.0.0.1 port 22 protocol ssh then restart
   if 3 restarts within 5 cycles then timeout              //设置在5个监控周期内重启3次则超时，那么就不再监控这个服务程序

check process httpd with pidfile /var/run/httpd.pid       //监控http服务
   start program = "/etc/init.d/httpd start"
   stop program = "/etc/init.d/httpd stop"
   if failed host 127.0.0.1 port 80 protocol http then restart
   if 5 restarts within 5 cycles then timeout

check process web_lb with pidfile /data/v20/server/web_lb/httpd.pid     //监控自定义服务
   start program = "/data/v20/bin/lb.sh"                 //启动脚本
   stop program = "/data/v20/bin/lb_stop.sh"         //停止脚本
   if failed host 10.10.8.2 port 16101 proto http then restart
   if failed host 10.10.8.2 port 16101 proto http for 5 times within 5 cycles then exec "/data/v20/bin/lb_pay.sh"
   if failed host 10.10.8.2 port 16102 type TCPSSL proto http then restart
   if failed host 10.10.8.2 port 16102 type TCPSSL proto http for 5 times within 5 cycles then exec "/data/v20/bin/lb_pay.sh

4）monit的启动（monit的默认端口是30000。最好在本地的/etc/hosts里面做下本机主机名的映射关系，将hostname映射到127.0.0.1）
[root@bastion-IDC src]# /etc/init.d/monit start/stop/reload/status/restart

[root@bastion-IDC ~]# monit -t                     //检测monit配置是否正确
[root@bastion-IDC ~]# monit reload              //重载monit配置
[root@bastion-IDC ~]# monit status             //查看monit进程监控情况

若启动monit的时候报错如下：

?
1
2
3
4
5
6
7
8
Cannot translate 'huanqiu_web2' to FQDN name -- Name or service not known
Generated unique Monit id af76cbce671f323782e09e0d114857fd and stored to '/root/.monit.id'
Reinitializing monit daemon
No daemon process found
 
解决办法：
在本机的/etc/hosts里面做下主机映射，即
127.0.0.1 huanqiu_web2

---------------------------------------------线上用过的一个配置------------------------------------------------

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
[root@huanqiu_web1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1 huanqiu_web1
  
[root@huanqiu_web1 ~]# cat /etc/monit.conf
set daemon 30
set logfile syslog facility log_daemon
set pidfile /var/run/monit.pid
set httpd port 30000
use address 127.0.0.1
allow 127.0.0.1
  
check process nginx with pidfile /Data/app/nginx/logs/nginx.pid
        start program = "/Data/app/nginx/sbin/nginx"
        stop program = "/Data/app/nginx/sbin/nginx -s stop"
  
check process php-fpm with pidfile /Data/app/php5.6.26/var/run/php-fpm.pid
       start program = "/Data/app/php5.6.26/sbin/php-fpm"
       stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep /Data/app/php5.6.26/etc/php-fpm.conf|grep -v grep|awk -F" " '{print $2}'`'"
 
check process mysql with pidfile /Data/app/mysql5.1.57/var/dev-new-test.pid
       start program = "/Data/app/mysql5.1.57/bin/mysqld_safe --defaults-file=/Data/app/mysql5.1.57/my.cnf &"
       stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep mysqld_safe|grep -v grep|awk -F" " '{print $2}'`'"
 
check process tomcat-7-admin-wls matching "/Data/app/tomcat-7-wls/conf"
        start program = "/Data/app/tomcat-7-wls/bin/startup.sh"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep /Data/app/tomcat-7-wls/conf|grep -v grep|awk -F" " '{print $2}'`'"
  
check process tomcat-7-wls matching "/Data/app/tomcat-7-wls/conf"
        start program = "/Data/app/tomcat-7-wls/bin/startup.sh"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep /Data/app/tomcat-7-wls/conf|grep -v grep|awk -F" " '{print $2}'`'"
  
check process tomcat-7 matching "/Data/app/tomcat-7/conf"
        start program = "/Data/app/tomcat-7/bin/startup.sh"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep /Data/app/tomcat-7/conf|grep -v grep|awk -F" " '{print $2}'`'"
 
check process tomcat-7-banshanbandao matching "/Data/app/tomcat-7-banshanbandao/conf"
        start program = "/Data/app/tomcat-7-banshanbandao/bin/startup.sh"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep /Data/app/tomcat-7-banshanbandao/conf|grep -v grep|awk -F" " '{print $2}'`'"
 
check process vpn matching "/etc/vpnc/vpnc-script"
        start program = "/bin/sh /bin/vpn_start"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep vpnc-script|grep -v grep|awk -F" " '{print $2}'`'"
 
[root@huanqiu_web1 ~]# monit -t
Control file syntax OK
[root@huanqiu_web1 ~]# /etc/init.d/monit start
Starting monit:                                            [  OK  ]
  
[root@huanqiu_web1 ~]# lsof -i:30000
COMMAND  PID USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
monit   6109 root    5u  IPv4 2438183462      0t0  TCP localhost:30000 (LISTEN)
 
[root@huanqiu_web1 ~]# monit reload
Reinitializing monit daemon
  
[root@huanqiu_web1 ~]# monit status
The Monit daemon 5.14 uptime: 8m
 
Process 'nginx'
  status                            Running
  monitoring status                 Monitored
  pid                               499
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            28d 20h 17m
  children                          8
  memory                            19.6 MB
  memory total                      381.6 MB
  memory percent                    0.0%
  memory percent total              0.5%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
Process 'php-fpm'
  status                            Running
  monitoring status                 Monitored
  pid                               3153
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            43d 19h 26m
  children                          16
  memory                            8.7 MB
  memory total                      352.3 MB
  memory percent                    0.0%
  memory percent total              0.5%
  cpu percent                       0.0%
  cpu percent total                 0.1%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
Process 'mysql'
  status                            Running
  monitoring status                 Monitored
  pid                               46403
  parent pid                        46254
  uid                               500
  effective uid                     500
  gid                               500
  uptime                            93d 0h 34m
  children                          0
  memory                            317.8 MB
  memory total                      317.8 MB
  memory percent                    0.4%
  memory percent total              0.4%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
Process 'tomcat-7-admin-wls'
  status                            Running
  monitoring status                 Monitored
  pid                               34188
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            4d 19h 15m
  children                          0
  memory                            803.6 MB
  memory total                      803.6 MB
  memory percent                    1.2%
  memory percent total              1.2%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
Process 'tomcat-7-wls'
  status                            Running
  monitoring status                 Monitored
  pid                               34188
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            4d 19h 15m
  children                          0
  memory                            803.6 MB
  memory total                      803.6 MB
  memory percent                    1.2%
  memory percent total              1.2%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
Process 'tomcat-7'
  status                            Running
  monitoring status                 Monitored
  pid                               14524
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            5d 21h 43m
  children                          0
  memory                            581.2 MB
  memory total                      581.2 MB
  memory percent                    0.9%
  memory percent total              0.9%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
Process 'tomcat-7-banshanbandao'
  status                            Running
  monitoring status                 Monitored
  pid                               29217
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            117d 0h 35m
  children                          0
  memory                            1.4 GB
  memory total                      1.4 GB
  memory percent                    2.1%
  memory percent total              2.1%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
Process 'vpn'
  status                            Running
  monitoring status                 Monitored
  pid                               13774
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            1h 36m
  children                          0
  memory                            2.4 MB
  memory total                      2.4 MB
  memory percent                    0.0%
  memory percent total              0.0%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 22 Mar 2017 11:32:42
 
System 'huanqiu_web1'
  status                            Running
  monitoring status                 Monitored
  load average                      [0.00] [0.04] [0.09]
  cpu                               0.6%us 0.1%sy 0.0%wa
  memory usage                      5.1 GB [8.0%]
  swap usage                        0 B [0.0%]
  data collected                    Wed, 22 Mar 2017 11:32:42

5）monit监控程序进程的方式
a）利用进程的pid文件进行监控：with pidfile
b）利用进程的关键字匹配方式进行监控: matching；可以使用“monit procmatch 进程名 CLI”来查找要匹配的唯一关键字
不管是pid文件里的pid号还是进程的关键字，都要求是唯一性的！必须是唯一的，如果matching匹配字段不唯一，那么监控无效！

下面罗列几个平时工作中常用的几个监控项：
[root@bastion-IDC ~]# cat /etc/monit.conf

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
set daemon 30
set logfile syslog facility log_daemon
set pidfile /var/run/monit.pid
set httpd port 30000
use address 127.0.0.1
allow 127.0.0.1
............
check process nginx with pidfile /usr/local/nginx/logs/nginx.pid
        start program = "/usr/local/nginx/sbin/nginx"
        stop program = "/usr/local/nginx/sbin/nginx -s stop"
 
check process nginx with pidfile /webserver/nginx/run/nginx.pid
        start program = "/webserver/init.d/nginx start" with timeout 10 seconds
        stop program = "/webserver/init.d/nginx stop"
        if failed host heylinux.com port 80 protocol http with timeout 10 seconds then restart
        if 3 restarts within 5 cycles then timeout group webserver
 
check process php-fpm with pidfile /var/run/php-fpm/php-fpm.pid
       start program = "/etc/init.d/php-fpm start"
       stop program = "/etc/init.d/php-fpm stop"
 
check process mysqld  with pidfile "/letv/mysql2/data/cdn.oss.letv.com.pid"
       start program = "/etc/init.d/mysqld start"
       stop program = "/etc/init.d/mysqld stop"
       if failed host 127.0.0.1 port 3306 then restart
 
check process mysql with pidfile /webserver/mysql/run/mysqld.pid
       start program = "/webserver/init.d/mysqld start" with timeout 10 seconds
       stop program = "/webserver/init.d/mysqld stop"
       if failed port 3307 protocol mysql with timeout 10 seconds then restart
       if 3 restarts within 5 cycles then timeout group webserver
 
check process memcached with pidfile "/var/run/memcached/memcached.pid"
       start program = "/etc/init.d/memcached start"
       stop program = "/etc/init.d/memcached stop"
       if failed host 127.0.0.1 port 11211 protocol memcache then restart
 
check process zabbix    with pidfile "/usr/local/zabbix/zabbix_agentd.pid"
        start program = "/usr/local/zabbix/sbin/zabbix_agentd -c /usr/local/zabbix/conf/zabbix_agentd.conf"
        stop program = "/bin/bash -c 'kill -s SIGTERM `cat /usr/local/zabbix/zabbix_agentd.pid`'"
        if failed host 127.0.0.1 port 10050 type tcp 2 times within 2 cycles then restart
 
check process httpd
        with pidfile "/usr/local/apache/logs/httpd.pid"
        start program = "/usr/local/apache/bin/httpd -k start"
        stop  program = "/bin/bash -c 'kill -s SIGTERM `cat /usr/local/apache/logs/httpd.pid`'"
 
check process redis
        with pidfile "/var/run/redis.pid"
        start program = "/usr/local/bin/redis-server /letv/uss/redis/redis.conf"
        stop  program = "/bin/bash -c 'kill -s SIGTERM `cat /var/run/redis.pid`'"
 
check process rsync with pidfile "/var/run/rsyncd.pid"
        start program = "/usr/bin/rsync --daemon"
        stop program = "/bin/bash -c 'kill -s SIGTERM `cat /var/run/rsyncd.pid`'"
 
check process pytask.py matching "/letv/p2sp/offline/pytask.py"
        start program = "/usr/bin/python /letv/p2sp/offline/pytask.py"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep offline/pytask.py|grep -v grep|awk -F" " '{print $2}'`'"
 
check process pytimed.py matching "/letv/p2sp/offline/pytimed.py"
        start program = "/usr/bin/python /letv/p2sp/offline/pytimed.py"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep offline/pytimed.py|grep -v grep|awk -F" " '{print $2}'`'"
 
check process hadoop with pidfile "/usr/local/hadoop/pids/hadoop-hadoop-datanode.pid"
     start program = "/usr/bin/sudo -u hadoop  -i  hadoop-daemon.sh start  datanode"
     stop program = "/usr/bin/sudo -u hadoop  -i  hadoop-daemon.sh stop  datanode"
 
check process ETMDaemon matching "/letv/p2sp/xware/lib/ETMDaemon"
        start program = "/letv/p2sp/xware/portal"
        stop program = "/bin/bash -c 'kill -s SIGTERM `ps -ef|grep ETMDaemon|grep -v grep |awk '{print $2}'`'"

如果监控配置项比较多，不想放在/etc/monit.conf文件里，那么可以定义include选项，比如：
[root@bastion-IDC ~]# vim /etc/monit.conf
.......
include /etc/services.cfg

然后创建/etc/services.cfg，将监控进程的配置集中放到这个文件里。

?
1
2
3
4
5
6
7
8
9
10
11
[root@bastion-IDC ~]# vim /etc/services.cfg
check process nginx with pidfile /usr/local/nginx/logs/nginx.pid
        start program = "/usr/local/nginx/sbin/nginx"
        stop program = "/usr/local/nginx/sbin/nginx -s stop"
check process php-fpm with pidfile /var/run/php-fpm/php-fpm.pid
       start program = "/etc/init.d/php-fpm start"
       stop program = "/etc/init.d/php-fpm stop"
check process mysqld  with pidfile "/letv/mysql2/data/cdn.oss.letv.com.pid"
       start program = "/etc/init.d/mysqld start"
       stop program = "/etc/init.d/mysqld stop"
.........

monit.conf文件里的邮件通知就可以这样配置：

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# mail-server
set mailserver  smtp.huanqiu.cn port 587
# email-format
set mail-format {
 from: monit@huanqiu.cn
 subject: $SERVICE $EVENT at $DATE on $HOST
 message: Monit $ACTION $SERVICE $EVENT at $DATE on $HOST : $DESCRIPTION.
 
       Yours sincerely,
          Monit
 
  }
 
set alert wangshibo@huanqiu.cn

Monit会提供几个内部变量（$DATE、$EVENT、$HOST等），你可以按照你的需求自定义邮件内容。如果你想要从Monit所在机器发送邮件，就需要一个已经安装的与sendmail兼容的程序（如postfix或者ssmtp）。

监控本机部分性能： 

?
1
2
3
4
5
6
check system 127.0.0.1
    if loadavg (5min) > 4 for 4 times 5 cycles then exec "/etc/monit/script/sendsms sysload 5min >4"
    if memory usage > 90% then exec "/etc/monit/script/sendsms 127.0.0.1 memory useage>90%"
    if cpu usage (user)  > 70% for 4 times within 5 cycles then exec "/etc/monit/script/sendsms cpu(user) >70%"
    if cpu usage (system) > 30% for 4 times within 5 cycles then exec "/etc/monit/script/sendsms cpu(system) >30% "
    if cpu usage (wait)  > 20% for 4 times within 5 cycles then exec "/etc/monit/script/sendsms system busy! cpu(wait) >20%"

监控远程机器的部分端口：

?
1
2
3
4
5
6
check host Unicom_mobi with address 211.90.246.51
      if failed icmp type echo count 10 with timeout 20 seconds then exec "/etc/monit/script/sendsms Unicom_mobi  211.90.246.51 ping failed!"
      if failed port 22 type tcp with timeout 10 seconds for 2 times within 3 cycles then exec "/etc/monit/script/sendsms unicom 211.90.246.51:2222 connect failed!"
      if failed port 9528 type tcp with timeout 10 seconds for 2 times within 3 cycles then exec "/etc/monit/script/sendsms unicom 211.90.246.51:9528 connect failed!"
      if failed port 9529 type tcp with timeout 10 seconds for 2 times within 3 cycles then exec "/etc/monit/script/sendsms unicom 211.90.246.51:9529 connect failed!"
      if failed port 9530 type tcp with timeout 10 seconds for 2 times within 3 cycles then exec "/etc/monit/script/sendsms unicom 211.90.246.51:9530 connect failed!"

monit好处是可以在监控故障设置重启服务和执行自定义脚本，如下

?
1
2
3
check filesystem root with path /dev/mapper/VolGroup00-LogVol00
      if space usage > 80% for 5 times within 15 cycles then exec "/etc/monit/script/clear_core.sh"
         else if succeed for 1 times within 2 cycles then exec "/etc/monit/script/sendsms '/dev/sda1 usage > 90% clear core file succeed!'>/dev/null 2"

再来看看几个小配置：

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
1）监控本地服务器的CPU、内存占用
check system localhost
    if loadavg (1min) > 10 then alert
    if loadavg (5min) > 6 then alert
    if memory usage > 75% then alert
    if cpu usage (user) > 70% then alert
    if cpu usage (system) > 60% then alert
    if cpu usage (wait) > 75% then alert
 
如果某个监控项不需要每个周期都检查，可以如下配置：
    if loadavg (1min) > 10 for 2 cycles then alert
 
2）设置一个检查远程SMTP服务器（如192.168.1.102）的监控。假定SMTP服务器运行着SMTP、IMAP、SSH服务。
check host MAIL with address 192.168.1.102
   if failed icmp type echo within 10 cycles then alert
   if failed port 25  protocol smtp then alert
      else if recovered then exec "/scripts/mail-script"
   if failed port 22  protocol ssh  then alert
   if failed port 143 protocol imap then alert
检查远程主机是否响应ICMP协议。如果我们在10个周期内没有收到ICMP回应，就发送一条报警。如果监测到25端口上的SMTP协议是异常的，就发送一条报警。如果在一次监测失败后又监测成功了，就运行一个脚本（/scripts/mail-script）。如果检查22端口上的SSH或者143端口上的IMAP协议不正常，同样发送报警。
 
 
3）存在性测试：
Monit当发现一个文件不存或者一个服务没有启动的时候默认操作是重启这个操作
check file with path /home/laicb/test.txt
   if does not exist for 5 cycles then alert
注意：
检测的是文件（使用的是path），如果只写了/home/laicb那么监视的时候就会提示，path不是一个有效的类型！
如果检测目录的话，就用directory替换path
 
4）资源测试：
有些资源可以在check system路口，有些可以在check entry路口，有些都可以。
if cpu is greater than 50% for 5 cycles then restart
注意：greater表示大于，%也可以用字节或者GB，MB等字符
 
5）时间戳测试：
时间戳[1]是指文件属性里的创建、修改、访问的时间。（下面的exec表示执行后面的命令动作）
改变形式：
 check file httpd.conf with path /usr/local/apache/conf/httpd.conf
   if changed timestamp
     then exec "/usr/local/apache/bin/apachectl graceful"
常量模式：
check file stored.ckp with path /msg-foo/config/stored.ckp
   if timestamp > 1 minute then alert
 
6）文件大小测试：
这个只能用在check file入口
check file with path /home/laicb/test.txt 
    if does not exist for 5 cycles then alert 
    if changed size for  1 cycles then alert            //如果没有指定，查看服务所对应的会发现是for 5 times within 5cycles  
 
如果更改文件大小，那么文件大小变化之后就在状态栏里显示size changed
 
 
7）权限测试：
check file monit.bin with path "/usr/local/bin/monit"
       if failed permission 0555 then unmonitor    //如果/usr/local/bin/monit文件权限不是555就拒绝执行
 
check file passwd with path /etc/passwd          
       if failed uid root then unmonitor    //如果不是root访问/etc/passwd那么拒绝访问
  
8）PID测试
 check process sshd with pidfile /var/run/sshd.pid
       if changed pid then exec "/my/script"
 
9）更新时间测试：
正常运行时间测试：
check process myapp with pidfile /var/run/myapp.pid
    start program = "/etc/init.d/myapp start"
    stop program = "/etc/init.d/myapp stop"
    if uptime > 3 days then restart
 
10）监控主机通信
 check host www.huanqiu.com with address www.huanqiu.com
       if failed icmp type echo count 5 with timeout 15 seconds
          then alert
 
11）apache程序监控
 check process apache with pidfile /var/run/httpd.pid
       start program = "/etc/init.d/httpd start"
       stop program  = "/etc/init.d/httpd stop"
       if cpu > 40% for 2 cycles then alert
       if totalcpu > 60% for 2 cycles then alert
       if totalcpu > 80% for 5 cycles then restart
       if mem > 100 MB for 5 cycles then stop
       if loadavg(5min) greater than 10.0 for 8 cycles then stop

-------------------------------------------------------------------------------------------
来看看下面遇到的几种monit不能使用的解决办法：
1）monit进程连接错误！（缺少http的端口支持，少了这部分内容）
最后经过排查发现，monit的配置文件/etc/monitrc里面少了下面两行：
set httpd port 30000
allow 127.0.0.1

将上面两行添加上，monit即可恢复正常使用状态中
[root@cdn ~]# cat /etc/monitrc
set daemon 30
set logfile syslog facility log_daemon
set pidfile /var/run/monit.pid
set httpd port 30000
allow 127.0.0.1
#allow admin:TVA3z3i
..........

2）另外一种错误：（修改127.0.0.1为localhost）
在monit -t和monit reload都没有报错的情况下，monit status报错如下：
[root@cdn ~]# monit status
monit: cannot read status from the monit daemon

查看日志信息：
[root@cdn ~]# tail -f /var/log/messages
Aug 18 19:27:21 cdn monit[14491]: monit: Denied connection from non-authorized client [220.181.153.243]
Aug 18 19:27:21 cdn monit[16899]: monit: cannot read status from the monit daemon

解决办法：
将monit配置文件中的“allow 127.0.0.1”修改为“allow localhost”即可！！
[root@cdn ~]# vim /etc/monitrc
set daemon 30
set logfile syslog facility log_daemon
set pidfile /var/run/monit.pid
set httpd port 30000
allow localhost
.........

这样，问题得到解决

[root@cdn ~]# monit status
The Monit daemon 5.3.2 uptime: 5m
.........................
Process 'nginx_down'
status Running
monitoring status Monitored
pid 18671
parent pid 1
uptime 145d 5h 17m
children 8
memory kilobytes 484
memory kilobytes total 4572
memory percent 0.0%
memory percent total 0.0%
cpu percent 0.0%
cpu percent total 0.0%
data collected Mon, 18 Aug 2014 19:34:25
.............

3）下面的错误在使用上面两种方法后，仍不能解决问题！ （添加use address 127.0.0.1）
[root@182 conf]# monit -t
Control file syntax OK
[root@182 conf]# monit reload
Reinitializing monit daemon
[root@182 conf]# monit status
monit: error connecting to the monit daemon

查看monit配置文件
[root@182 conf]# cat /etc/monitrc.bak
set daemon 30
set httpd port 30000
allow 127.0.0.1
set logfile syslog facility log_daemon
set pidfile /var/run/monit.pid
..................

最后解决办法：
需要在monit配置文件中添加“use address 127.0.0.1"内容！
[root@182 conf]# cat /etc/monitrc
set daemon 30
set httpd port 30000
use address 127.0.0.1
allow 127.0.0.1
set logfile syslog facility log_daemon
set pidfile /var/run/monit.pid
.............

查看，问题已经得到解决
[root@182 conf]# monit status
The Monit daemon 5.3.2 uptime: 7m
..............
Process 'rsync'
status Running
monitoring status Monitored
pid 13519
parent pid 1
uptime 393d 9h 8m
children 0
memory kilobytes 540
memory kilobytes total 540
memory percent 0.0%
memory percent total 0.0%
cpu percent 0.0%
cpu percent total 0.0%
data collected Thu, 21 Aug 2014 19:24:58
..............

注意：
上面的第3钟方式是最全面的，如果添加了use address 127.0.0.1后，使用monit status仍然出现下面的情况：
[root@ly-u-gfs1 ~]# monit status
monit: error connecting to the monit daemon

那么就稍微等待一会儿，等一小段时间后，就会发现monit使用顺畅了
[root@ly-u-gfs1 ~]# monit status
The Monit daemon 5.3.2 uptime: 6m

Process 'net-snmp'
status Running
monitoring status Monitored

-------------------------------------------------------------------------------------------------------------------------
Centos7下部署Monit环境过程：
[root@linux-node2 ~]# yum update
[root@linux-node2 ~]# yum install -y monit
[root@linux-node2 ~]# rpm -ql monit
/etc/logrotate.d/monit
/etc/monit.d
/etc/monit.d/logging
/etc/monitrc
/usr/bin/monit
/usr/lib/systemd/system/monit.service
/usr/share/doc/monit-5.14
/usr/share/doc/monit-5.14/COPYING
/usr/share/doc/monit-5.14/README
/usr/share/man/man1/monit.1.gz
/var/log/monit.log
[root@linux-node2 ~]# monit -V
This is Monit version 5.14
Copyright (C) 2001-2016 Tildeslash Ltd. All Rights Reserved.

[root@linux-node2 ~]# monit -v
Adding host allow 'localhost'
Skipping redundant host 'localhost'
Adding credentials for user 'admin'
Runtime constants:
Control file = /etc/monitrc
Log file = /var/log/monit.log
Pid file = /run/monit.pid
Id file = /root/.monit.id
State file = /root/.monit.state
Debug = True
Log = True
Use syslog = False
Is Daemon = True
Use process engine = True
Poll time = 30 seconds with start delay 0 seconds
Expect buffer = 256 bytes
Mail from = (not defined)
Mail subject = (not defined)
Mail message = (not defined)
Start monit httpd = True
httpd bind address = localhost
httpd portnumber = 2812
httpd ssl = Disabled
httpd signature = Enabled
httpd auth. style = Basic Authentication and Host/Net allow list

The service list contains the following entries:

System Name = linux-node2.openstack
Monitoring mode = active

查看默认的配置文件内容
[root@linux-node2 ~]# grep -v '^#' /etc/monitrc
set daemon 30 # check services at 30 seconds intervals
set logfile syslog

set httpd port 2812 and
use address localhost         # only accept connection from localhost
allow localhost                   # allow localhost to connect to the server and
allow admin:monit              # require user 'admin' with password 'monit'
allow @monit                    # allow users of group 'monit' to connect (rw)
allow @users readonly       # allow users of group 'users' to connect readonly

include /etc/monit.d/*

[root@linux-node2 ~]# cat /etc/logrotate.d/monit
/var/log/monit.log {
missingok
notifempty
size 100k
create 0644 root root
postrotate
/bin/systemctl reload monit.service > /dev/null 2>&1 || :
endscript
}

[root@linux-node2 ~]# cat /etc/monit.d/logging
# log to monit.log
set logfile /var/log/monit.log                     //监视周期为60秒，日志输出及日志滚动以配置好了

配置monit
[root@linux-node2 ~]# vim /etc/monitrc

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
set daemon  5  
set logfile syslog
  
set httpd port 2812 and
    use address localhost
    allow localhost      
    allow admin:monit    
    allow @monit         
    allow @users readonly   
  
include /etc/monit.d/*
  
check process sshd with pidfile /var/run/sshd.pid
    start program "/usr/bin/systemctl start sshd.service"
    stop program "/usr/bin/systemctl stop sshd.service"
    if failed port 22 protocol ssh then restart
    if 5 restart within 5 cycles then timeout
  
check process apache with pidfile /etc/httpd/run/httpd.pid
  start program = "/usr/bin/systemctl start httpd" with timeout 60 seconds
  stop program  = "/usr/bin/systemctl stop httpd"
  if failed host linux-node2.openstack port 80 protocol http
     and request "/readme.html"
     then restart
  if 3 restarts within 5 cycles then timeout
  group apache
  
check process mariadb with pidfile "/var/lib/mysql/linux-node2.pid"
    start = "/usr/bin/systemctl start mariadb.service"
    stop = "/usr/bin/systemctl stop mariadb.service"
    if failed host 127.0.0.1 port 3306 protocol mysql then restart
    if 5 restarts within 5 cycles then timeout

----------------------------------------------------------------------------------------------
查看mysql服务的pid
MariaDB [(none)]> show variables like "%pid%";
+---------------+--------------------------------+
| Variable_name | Value |
+---------------+--------------------------------+
| pid_file | /var/lib/mysql/linux-node2.pid |
+---------------+--------------------------------+
1 row in set (0.00 sec)
--------------------------------------------------------------------------------------------

启动monit
[root@linux-node2 ~]# systemctl enable monit.service
Created symlink from /etc/systemd/system/multi-user.target.wants/monit.service to /usr/lib/systemd/system/monit.service.
[root@linux-node2 ~]# systemctl start monit.service
[root@linux-node2 ~]# lsof -i:2812
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
monit 89106 root 5u IPv4 172788270 0t0 TCP localhost:atmtcp (LISTEN)
[root@linux-node2 ~]# systemctl status monit.service
● monit.service - Pro-active monitoring utility for unix systems
Loaded: loaded (/usr/lib/systemd/system/monit.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2017-02-03 10:47:22 CST; 50s ago
Main PID: 89106 (monit)
CGroup: /system.slice/monit.service
└─89106 /usr/bin/monit -I

Feb 03 10:47:22 linux-node2.openstack systemd[1]: Started Pro-active monitoring utility for unix systems.
Feb 03 10:47:23 linux-node2.openstack systemd[1]: Starting Pro-active monitoring utility for unix systems...
Feb 03 10:47:23 linux-node2.openstack monit[89106]: /etc/monitrc:20: Program does not exist: 'systemctl'
Feb 03 10:47:23 linux-node2.openstack monit[89106]: /etc/monitrc:21: Program does not exist: 'systemctl'
Feb 03 10:47:23 linux-node2.openstack monit[89106]: Starting Monit 5.14 daemon with http interface at [localhost]:2812

查看monit状态

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
[root@linux-node2 ~]# monit status
The Monit daemon 5.14 uptime: 9m
 
Process 'sshd'
  status                            Running
  monitoring status                 Monitored
  pid                               1755
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            86d 19h 39m
  children                          6
  memory                            3.5 MB
  memory total                      25.1 MB
  memory percent                    0.0%
  memory percent total              0.0%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  port response time                0.021s to [localhost]:22 type TCP/IP protocol SSH
  data collected                    Fri, 03 Feb 2017 10:57:20
 
Process 'apache'
  status                            Not monitored
  monitoring status                 Not monitored
  data collected                    Fri, 03 Feb 2017 10:50:21
 
Process 'mariadb'
  status                            Running
  monitoring status                 Monitored
  pid                               46235
  parent pid                        1
  uid                               27
  effective uid                     27
  gid                               27
  uptime                            29d 16h 1m
  children                          0
  memory                            296.1 MB
  memory total                      296.1 MB
  memory percent                    0.4%
  memory percent total              0.4%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  port response time                0.001s to [127.0.0.1]:3306 type TCP/IP protocol MYSQL
  data collected                    Fri, 03 Feb 2017 10:57:20
 
System 'linux-node2.openstack'
  status                            Running
  monitoring status                 Monitored
  load average                      [2.01] [1.86] [1.94]
  cpu                               5.2%us 2.0%sy 0.0%wa
  memory usage                      44.0 GB [70.1%]
  swap usage                        2.6 MB [0.1%]
  data collected                    Fri, 03 Feb 2017 10:57:20

重新载入monit服务
[root@linux-node2 ~]# monit reload
Reinitializing monit daemon

确认monit自动启动进程
停止nginx进程之后，查看monit.log文件
[root@linux-node2 ~]# systemctl stop nginx.service
[root@linux-node2 ~]# tailf /var/log/monit.log
[CST Apr 5 21:35:18] error : 'nginx' process is not running
[CST Apr 5 21:35:18] info : 'nginx' trying to restart
[CST Apr 5 21:35:18] info : 'nginx' start: /usr/bin/systemctl

配置启动启动。根据系统及版本自动启动的命令不同，在这里介绍CentOS7上配置自动启动的方法
[root@linux-node2 ~]# systemctl list-unit-files | grep monit.service
monit.service disabled
[root@linux-node2 ~]# systemctl enable monit.service
ln -s '/usr/lib/systemd/system/monit.service' '/etc/systemd/system/multi-user.target.wants/monit.service'
[root@linux-node2 ~]# systemctl list-unit-files | grep monit.service
monit.service enabled
-------------------------------------------------------------------------------------------------------------------------

posted @   散尽浮华  阅读(11011)  评论(1)  编辑  收藏  举报
编辑推荐：
· 从 HTTP 原因短语缺失研究 HTTP/2 和 HTTP/3 的设计差异
· AI与.NET技术实操系列：向量存储与相似性搜索在 .NET 中的实现
· 基于Microsoft.Extensions.AI核心库实现RAG应用
· Linux系列：如何用heaptrack跟踪.NET程序的非托管内存泄露
· 开发者必知的日志记录最佳实践
阅读排行：
· winform 绘制太阳，地球，月球 运作规律
· AI与.NET技术实操系列（五）：向量存储与相似性搜索在 .NET 中的实现
· 超详细：普通电脑也行Windows部署deepseek R1训练数据并当服务器共享给他人
· 【硬核科普】Trae如何「偷看」你的代码？零基础破解AI编程运行原理
· 上周热点回顾（3.3-3.9）
点击右上角即可分享
微信分享提示
AI IDE Trae