MHA常用命令和监控告警配置测试

==========================================
[server default]
user = root
password = root654321
ssh_user = root
repl_user = rep
repl_password = rep
ping_interval = 1
ping_type = SELECT

manager_workdir=/etc/masterha/app1
manager_log=/var/log/masterha/app1/manager.log
remote_workdir=/tmp

master_ip_failover_script="/usr/local/scripts/master_ip_failover"
master_ip_online_change_script="/usr/local/scripts/master_ip_online_change"

shutdown_script=""
report_script="/usr/local/send_report"
check_repl_delay=0

[server1]
hostname=192.168.55.108
port=5566
master_binlog_dir="/data/my5566/log/"
no_master=1

[server2]
hostname=192.168.55.105
port=5566
master_binlog_dir="/data/my5566/log/"
candidate_master=1

[server3]
hostname=192.168.55.106
port=5566
master_binlog_dir="/data/my5566/log/"
candidate_master=1

===================================================

[root@host-192-168-55-103 ~]# masterha_check_ssh --conf=/etc/masterha/app1/app1.conf
Wed May 19 16:23:50 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed May 19 16:23:50 2021 - [info] Reading application default configuration from /etc/masterha/app1/app1.conf..
Wed May 19 16:23:50 2021 - [info] Reading server configuration from /etc/masterha/app1/app1.conf..
Wed May 19 16:23:50 2021 - [info] Starting SSH connection tests..
Wed May 19 16:23:50 2021 - [debug]
Wed May 19 16:23:50 2021 - [debug] Connecting via SSH from root@192.168.55.108(192.168.55.108:22) to root@192.168.55.105(192.168.55.105:22)..
Wed May 19 16:23:50 2021 - [debug] ok.
Wed May 19 16:23:50 2021 - [debug] Connecting via SSH from root@192.168.55.108(192.168.55.108:22) to root@192.168.55.106(192.168.55.106:22)..
Wed May 19 16:23:50 2021 - [debug] ok.
Wed May 19 16:23:51 2021 - [debug]
Wed May 19 16:23:50 2021 - [debug] Connecting via SSH from root@192.168.55.105(192.168.55.105:22) to root@192.168.55.108(192.168.55.108:22)..
Wed May 19 16:23:50 2021 - [debug] ok.
Wed May 19 16:23:50 2021 - [debug] Connecting via SSH from root@192.168.55.105(192.168.55.105:22) to root@192.168.55.106(192.168.55.106:22)..
Wed May 19 16:23:50 2021 - [debug] ok.
Wed May 19 16:23:51 2021 - [debug]
Wed May 19 16:23:51 2021 - [debug] Connecting via SSH from root@192.168.55.106(192.168.55.106:22) to root@192.168.55.108(192.168.55.108:22)..
Wed May 19 16:23:51 2021 - [debug] ok.
Wed May 19 16:23:51 2021 - [debug] Connecting via SSH from root@192.168.55.106(192.168.55.106:22) to root@192.168.55.105(192.168.55.105:22)..
Wed May 19 16:23:51 2021 - [debug] ok.
Wed May 19 16:23:51 2021 - [info] All SSH connection tests passed successfully.
Use of uninitialized value in exit at /usr/local/bin/masterha_check_ssh line 44.
[root@host-192-168-55-103 ~]#

 

[root@host-192-168-55-103 ~]# masterha_check_repl --conf=/etc/masterha/app1/app1.conf
Wed May 19 16:38:31 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed May 19 16:38:31 2021 - [info] Reading application default configuration from /etc/masterha/app1/app1.conf..
Wed May 19 16:38:31 2021 - [info] Reading server configuration from /etc/masterha/app1/app1.conf..
Wed May 19 16:38:31 2021 - [info] MHA::MasterMonitor version 0.58.
Wed May 19 16:38:32 2021 - [info] GTID failover mode = 1
Wed May 19 16:38:32 2021 - [info] Dead Servers:
Wed May 19 16:38:32 2021 - [info] Alive Servers:
Wed May 19 16:38:32 2021 - [info] 192.168.55.108(192.168.55.108:5566)
Wed May 19 16:38:32 2021 - [info] 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:38:32 2021 - [info] 192.168.55.106(192.168.55.106:5566)
Wed May 19 16:38:32 2021 - [info] Alive Slaves:
Wed May 19 16:38:32 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 16:38:32 2021 - [info] GTID ON
Wed May 19 16:38:32 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:38:32 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 16:38:32 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 16:38:32 2021 - [info] GTID ON
Wed May 19 16:38:32 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:38:32 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 16:38:32 2021 - [info] Current Alive Master: 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:38:32 2021 - [info] Checking slave configurations..
Wed May 19 16:38:32 2021 - [info] Checking replication filtering settings..
Wed May 19 16:38:32 2021 - [info] binlog_do_db= , binlog_ignore_db=
Wed May 19 16:38:32 2021 - [info] Replication filtering check ok.
Wed May 19 16:38:32 2021 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Wed May 19 16:38:32 2021 - [info] Checking SSH publickey authentication settings on the current master..
Wed May 19 16:38:32 2021 - [info] HealthCheck: SSH to 192.168.55.105 is reachable.
Wed May 19 16:38:32 2021 - [info]
192.168.55.105(192.168.55.105:5566) (current master)
+--192.168.55.108(192.168.55.108:5566)
+--192.168.55.106(192.168.55.106:5566)

Wed May 19 16:38:32 2021 - [info] Checking replication health on 192.168.55.108..
Wed May 19 16:38:32 2021 - [info] ok.
Wed May 19 16:38:32 2021 - [info] Checking replication health on 192.168.55.106..
Wed May 19 16:38:32 2021 - [info] ok.
Wed May 19 16:38:32 2021 - [info] Checking master_ip_failover_script status:
Wed May 19 16:38:32 2021 - [info] /usr/local/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.55.105 --orig_master_ip=192.168.55.105 --orig_master_port=5566
Wed May 19 16:38:32 2021 - [info] OK.
Wed May 19 16:38:32 2021 - [warning] shutdown_script is not defined.
Wed May 19 16:38:32 2021 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

 

首次给mysql master主机上添加VIP
ip addr add 192.168.55.200/24 dev eth0


启动MHA manager
nohup masterha_manager --conf=/etc/masterha/app1/app1.conf --ignore_last_failover &


启动后查看日志
[root@host-192-168-55-103 app1]# pwd
/var/log/masterha/app1
[root@host-192-168-55-103 app1]# cat manager.log
Wed May 19 16:42:55 2021 - [info] MHA::MasterMonitor version 0.58.
Wed May 19 16:42:56 2021 - [info] GTID failover mode = 1
Wed May 19 16:42:56 2021 - [info] Dead Servers:
Wed May 19 16:42:56 2021 - [info] Alive Servers:
Wed May 19 16:42:56 2021 - [info] 192.168.55.108(192.168.55.108:5566)
Wed May 19 16:42:56 2021 - [info] 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:42:56 2021 - [info] 192.168.55.106(192.168.55.106:5566)
Wed May 19 16:42:56 2021 - [info] Alive Slaves:
Wed May 19 16:42:56 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 16:42:56 2021 - [info] GTID ON
Wed May 19 16:42:56 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:42:56 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 16:42:56 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 16:42:56 2021 - [info] GTID ON
Wed May 19 16:42:56 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:42:56 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 16:42:56 2021 - [info] Current Alive Master: 192.168.55.105(192.168.55.105:5566)
Wed May 19 16:42:56 2021 - [info] Checking slave configurations..
Wed May 19 16:42:56 2021 - [info] Checking replication filtering settings..
Wed May 19 16:42:56 2021 - [info] binlog_do_db= , binlog_ignore_db=
Wed May 19 16:42:56 2021 - [info] Replication filtering check ok.
Wed May 19 16:42:56 2021 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Wed May 19 16:42:56 2021 - [info] Checking SSH publickey authentication settings on the current master..
Wed May 19 16:42:56 2021 - [info] HealthCheck: SSH to 192.168.55.105 is reachable.
Wed May 19 16:42:56 2021 - [info]
192.168.55.105(192.168.55.105:5566) (current master)
+--192.168.55.108(192.168.55.108:5566)
+--192.168.55.106(192.168.55.106:5566)

Wed May 19 16:42:56 2021 - [info] Checking master_ip_failover_script status:
Wed May 19 16:42:56 2021 - [info] /usr/local/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.55.105 --orig_master_ip=192.168.55.105 --orig_master_port=5566
Wed May 19 16:42:57 2021 - [info] OK.
Wed May 19 16:42:57 2021 - [warning] shutdown_script is not defined.
Wed May 19 16:42:57 2021 - [info] Set master ping interval 1 seconds.
Wed May 19 16:42:57 2021 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Wed May 19 16:42:57 2021 - [info] Starting ping health check on 192.168.55.105(192.168.55.105:5566)..
Wed May 19 16:42:57 2021 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
[root@host-192-168-55-103 app1]#


检查MHA状态
[root@host-192-168-55-103 app1]# masterha_check_status --conf=/etc/masterha/app1/app1.conf
app1 (pid:30508) is running(0:PING_OK), master:192.168.55.105
[root@host-192-168-55-103 app1]#

 

关闭mysql master实例 failover验证：
[root@host-192-168-55-105 ~]# ps -ef | grep mysqld
root 29564 29385 0 14:55 pts/0 00:00:00 /bin/sh /usr/local/mysql5724/bin/mysqld_safe --defaults-file=/etc/mysql/my5566.cnf --user=mysql
mysql 30855 29564 0 14:55 pts/0 00:00:17 /usr/local/mysql5724/bin/mysqld --defaults-file=/etc/mysql/my5566.cnf --basedir=/usr/local/mysql5724/ --datadir=/data/my5566/data --plugin-dir=/usr/local/mysql5724//lib/mysql/plugin --user=mysql --log-error=/data/my5566/log/error.log --open-files-limit=68192 --pid-file=/data/my5566/socket/mysql.pid --socket=/data/my5566/socket/mysqld.sock --port=5566
root 31849 29385 0 17:04 pts/0 00:00:00 grep --color=auto mysqld
[root@host-192-168-55-105 ~]# /usr/local/mysql5724/bin/mysqladmin shutdown
[1]+ Done nohup /usr/local/mysql5724/bin/mysqld_safe --defaults-file=/etc/mysql/my5566.cnf --user=mysql
[root@host-192-168-55-105 ~]#


Wed May 19 17:04:23 2021 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Wed May 19 17:04:23 2021 - [info] Executing SSH check script: exit 0
Wed May 19 17:04:23 2021 - [info] HealthCheck: SSH to 192.168.55.105 is reachable.
Wed May 19 17:04:24 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.55.105' (111))
Wed May 19 17:04:24 2021 - [warning] Connection failed 2 time(s)..
Wed May 19 17:04:25 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.55.105' (111))
Wed May 19 17:04:25 2021 - [warning] Connection failed 3 time(s)..
Wed May 19 17:04:26 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.55.105' (111))
Wed May 19 17:04:26 2021 - [warning] Connection failed 4 time(s)..
Wed May 19 17:04:26 2021 - [warning] Master is not reachable from health checker!
Wed May 19 17:04:26 2021 - [warning] Master 192.168.55.105(192.168.55.105:5566) is not reachable!
Wed May 19 17:04:26 2021 - [warning] SSH is reachable.
Wed May 19 17:04:26 2021 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1/app1.conf again, and trying to connect to all servers to check server status..
Wed May 19 17:04:26 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed May 19 17:04:26 2021 - [info] Reading application default configuration from /etc/masterha/app1/app1.conf..
Wed May 19 17:04:26 2021 - [info] Reading server configuration from /etc/masterha/app1/app1.conf..
Wed May 19 17:04:27 2021 - [info] GTID failover mode = 1
Wed May 19 17:04:27 2021 - [info] Dead Servers:
Wed May 19 17:04:27 2021 - [info] 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:27 2021 - [info] Alive Servers:
Wed May 19 17:04:27 2021 - [info] 192.168.55.108(192.168.55.108:5566)
Wed May 19 17:04:27 2021 - [info] 192.168.55.106(192.168.55.106:5566)
Wed May 19 17:04:27 2021 - [info] Alive Slaves:
Wed May 19 17:04:27 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:27 2021 - [info] GTID ON
Wed May 19 17:04:27 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:27 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 17:04:27 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:27 2021 - [info] GTID ON
Wed May 19 17:04:27 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:27 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 17:04:27 2021 - [info] Checking slave configurations..
Wed May 19 17:04:27 2021 - [info] Checking replication filtering settings..
Wed May 19 17:04:27 2021 - [info] Replication filtering check ok.
Wed May 19 17:04:27 2021 - [info] Master is down!
Wed May 19 17:04:27 2021 - [info] Terminating monitoring script.
Wed May 19 17:04:27 2021 - [info] Got exit code 20 (Master dead).
Wed May 19 17:04:27 2021 - [info] MHA::MasterFailover version 0.58.
Wed May 19 17:04:27 2021 - [info] Starting master failover.
Wed May 19 17:04:27 2021 - [info]
Wed May 19 17:04:27 2021 - [info] * Phase 1: Configuration Check Phase..
Wed May 19 17:04:27 2021 - [info]
Wed May 19 17:04:28 2021 - [info] GTID failover mode = 1
Wed May 19 17:04:28 2021 - [info] Dead Servers:
Wed May 19 17:04:28 2021 - [info] 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Checking master reachability via MySQL(double check)...
Wed May 19 17:04:28 2021 - [info] ok.
Wed May 19 17:04:28 2021 - [info] Alive Servers:
Wed May 19 17:04:28 2021 - [info] 192.168.55.108(192.168.55.108:5566)
Wed May 19 17:04:28 2021 - [info] 192.168.55.106(192.168.55.106:5566)
Wed May 19 17:04:28 2021 - [info] Alive Slaves:
Wed May 19 17:04:28 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 17:04:28 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 17:04:28 2021 - [info] Starting GTID based failover.
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] ** Phase 1: Configuration Check Phase completed.
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] * Phase 2: Dead Master Shutdown Phase..
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] Forcing shutdown so that applications never connect to the current master..
Wed May 19 17:04:28 2021 - [info] Executing master IP deactivation script:
Wed May 19 17:04:28 2021 - [info] /usr/local/scripts/master_ip_failover --orig_master_host=192.168.55.105 --orig_master_ip=192.168.55.105 --orig_master_port=5566 --command=stopssh --ssh_user=root


IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.55.200/24===

Disabling the VIP on old master: 192.168.55.105
SIOCSIFFLAGS: Cannot assign requested address
Wed May 19 17:04:28 2021 - [info] done.
Wed May 19 17:04:28 2021 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Wed May 19 17:04:28 2021 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] * Phase 3: Master Recovery Phase..
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] The latest binary log file/position on all slaves is mysql-bin.000002:1156
Wed May 19 17:04:28 2021 - [info] Retrieved Gtid Set: b920d1d9-b86d-11eb-86b1-fa163ea639a1:1-5
Wed May 19 17:04:28 2021 - [info] Latest slaves (Slaves that received relay log files to the latest):
Wed May 19 17:04:28 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 17:04:28 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 17:04:28 2021 - [info] The oldest binary log file/position on all slaves is mysql-bin.000002:1156
Wed May 19 17:04:28 2021 - [info] Retrieved Gtid Set: b920d1d9-b86d-11eb-86b1-fa163ea639a1:1-5
Wed May 19 17:04:28 2021 - [info] Oldest slaves:
Wed May 19 17:04:28 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 17:04:28 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] * Phase 3.3: Determining New Master Phase..
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] Searching new master from slaves..
Wed May 19 17:04:28 2021 - [info] Candidate masters from the configuration file:
Wed May 19 17:04:28 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 17:04:28 2021 - [info] Non-candidate masters:
Wed May 19 17:04:28 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 17:04:28 2021 - [info] GTID ON
Wed May 19 17:04:28 2021 - [info] Replicating from 192.168.55.105(192.168.55.105:5566)
Wed May 19 17:04:28 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 17:04:28 2021 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Wed May 19 17:04:28 2021 - [info] New master is 192.168.55.106(192.168.55.106:5566)
Wed May 19 17:04:28 2021 - [info] Starting master failover..
Wed May 19 17:04:28 2021 - [info]
From:
192.168.55.105(192.168.55.105:5566) (current master)
+--192.168.55.108(192.168.55.108:5566)
+--192.168.55.106(192.168.55.106:5566)

To:
192.168.55.106(192.168.55.106:5566) (new master)
+--192.168.55.108(192.168.55.108:5566)
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] * Phase 3.3: New Master Recovery Phase..
Wed May 19 17:04:28 2021 - [info]
Wed May 19 17:04:28 2021 - [info] Waiting all logs to be applied..
Wed May 19 17:04:28 2021 - [info] done.
Wed May 19 17:04:28 2021 - [info] Replicating from the latest slave 192.168.55.108(192.168.55.108:5566) and waiting to apply..
Wed May 19 17:04:28 2021 - [info] Waiting all logs to be applied on the latest slave..
Wed May 19 17:04:28 2021 - [info] Resetting slave 192.168.55.106(192.168.55.106:5566) and starting replication from the new master 192.168.55.108(192.168.55.108:5566)..
Wed May 19 17:04:29 2021 - [info] Executed CHANGE MASTER.
Wed May 19 17:04:30 2021 - [info] Slave started.
Wed May 19 17:04:30 2021 - [info] Waiting to execute all relay logs on 192.168.55.106(192.168.55.106:5566)..
Wed May 19 17:04:30 2021 - [info] master_pos_wait(mysql-bin.000002:1552) completed on 192.168.55.106(192.168.55.106:5566). Executed 0 events.
Wed May 19 17:04:30 2021 - [info] done.
Wed May 19 17:04:30 2021 - [info] done.
Wed May 19 17:04:30 2021 - [info] Getting new master's binlog name and position..
Wed May 19 17:04:30 2021 - [info] mysql-bin.000002:1948
Wed May 19 17:04:30 2021 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.55.106', MASTER_PORT=5566, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Wed May 19 17:04:30 2021 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000002, 1948, 3e5d6f2d-b86f-11eb-a7b7-fa163ebdc88f:1-2,
b920d1d9-b86d-11eb-86b1-fa163ea639a1:1-5,
cd7a22f5-b86d-11eb-b9ec-fa163ecf4f93:1-2
Wed May 19 17:04:30 2021 - [info] Executing master IP activate script:
Wed May 19 17:04:30 2021 - [info] /usr/local/scripts/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.55.105 --orig_master_ip=192.168.55.105 --orig_master_port=5566 --new_master_host=192.168.55.106 --new_master_ip=192.168.55.106 --new_master_port=5566 --new_master_user='root' --new_master_password=xxx
Unknown option: new_master_user
Unknown option: new_master_password


IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.55.200/24===

Enabling the VIP - 192.168.55.200/24 on the new master - 192.168.55.106
Wed May 19 17:04:32 2021 - [info] OK.
Wed May 19 17:04:32 2021 - [info] Setting read_only=0 on 192.168.55.106(192.168.55.106:5566)..
Wed May 19 17:04:32 2021 - [info] ok.
Wed May 19 17:04:32 2021 - [info] ** Finished master recovery successfully.
Wed May 19 17:04:32 2021 - [info] * Phase 3: Master Recovery Phase completed.
Wed May 19 17:04:32 2021 - [info]
Wed May 19 17:04:32 2021 - [info] * Phase 4: Slaves Recovery Phase..
Wed May 19 17:04:32 2021 - [info]
Wed May 19 17:04:32 2021 - [info]
Wed May 19 17:04:32 2021 - [info] * Phase 4.1: Starting Slaves in parallel..
Wed May 19 17:04:32 2021 - [info]
Wed May 19 17:04:32 2021 - [info] -- Slave recovery on host 192.168.55.108(192.168.55.108:5566) started, pid: 527. Check tmp log /etc/masterha/app1/192.168.55.108_5566_20210519170427.log if it takes time..
Wed May 19 17:04:33 2021 - [info]
Wed May 19 17:04:33 2021 - [info] Log messages from 192.168.55.108 ...
Wed May 19 17:04:33 2021 - [info]
Wed May 19 17:04:32 2021 - [info] Resetting slave 192.168.55.108(192.168.55.108:5566) and starting replication from the new master 192.168.55.106(192.168.55.106:5566)..
Wed May 19 17:04:32 2021 - [info] Executed CHANGE MASTER.
Wed May 19 17:04:32 2021 - [info] Slave started.
Wed May 19 17:04:32 2021 - [info] gtid_wait(3e5d6f2d-b86f-11eb-a7b7-fa163ebdc88f:1-2,
b920d1d9-b86d-11eb-86b1-fa163ea639a1:1-5,
cd7a22f5-b86d-11eb-b9ec-fa163ecf4f93:1-2) completed on 192.168.55.108(192.168.55.108:5566). Executed 3 events.
Wed May 19 17:04:33 2021 - [info] End of log messages from 192.168.55.108.
Wed May 19 17:04:33 2021 - [info] -- Slave on host 192.168.55.108(192.168.55.108:5566) started.
Wed May 19 17:04:33 2021 - [info] All new slave servers recovered successfully.
Wed May 19 17:04:33 2021 - [info]
Wed May 19 17:04:33 2021 - [info] * Phase 5: New master cleanup phase..
Wed May 19 17:04:33 2021 - [info]
Wed May 19 17:04:33 2021 - [info] Resetting slave info on the new master..
Wed May 19 17:04:33 2021 - [info] 192.168.55.106: Resetting slave info succeeded.
Wed May 19 17:04:33 2021 - [info] Master failover to 192.168.55.106(192.168.55.106:5566) completed successfully.
Wed May 19 17:04:33 2021 - [info]

----- Failover Report -----

app1: MySQL Master failover 192.168.55.105(192.168.55.105:5566) to 192.168.55.106(192.168.55.106:5566) succeeded

Master 192.168.55.105(192.168.55.105:5566) is down!

Check MHA Manager logs at host-192-168-55-103:/var/log/masterha/app1/manager.log for details.

Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.55.105(192.168.55.105:5566)
Selected 192.168.55.106(192.168.55.106:5566) as a new master.
192.168.55.106(192.168.55.106:5566): OK: Applying all logs succeeded.
192.168.55.106(192.168.55.106:5566): OK: Activated master IP address.
192.168.55.108(192.168.55.108:5566): OK: Slave started, replicating from 192.168.55.106(192.168.55.106:5566)
192.168.55.106(192.168.55.106:5566): Resetting slave info succeeded.
Master failover to 192.168.55.106(192.168.55.106:5566) completed successfully.

 

在线手动切换主从（切换失败）。。。
[root@host-192-168-55-103 app1]# masterha_master_switch --master_state=alive --conf=/etc/masterha/app1/app1.conf --new_master_host=192.168.55.105 --orig_master_is_new_slave
Wed May 19 18:09:49 2021 - [info] MHA::MasterRotate version 0.58.
Wed May 19 18:09:49 2021 - [info] Starting online master switch..
Wed May 19 18:09:49 2021 - [info]
Wed May 19 18:09:49 2021 - [info] * Phase 1: Configuration Check Phase..
Wed May 19 18:09:49 2021 - [info]
Wed May 19 18:09:49 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed May 19 18:09:49 2021 - [info] Reading application default configuration from /etc/masterha/app1/app1.conf..
Wed May 19 18:09:49 2021 - [info] Reading server configuration from /etc/masterha/app1/app1.conf..
Wed May 19 18:09:50 2021 - [info] GTID failover mode = 1
Wed May 19 18:09:50 2021 - [info] Current Alive Master: 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:09:50 2021 - [info] Alive Slaves:
Wed May 19 18:09:50 2021 - [info] 192.168.55.105(192.168.55.105:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 18:09:50 2021 - [info] GTID ON
Wed May 19 18:09:50 2021 - [info] Replicating from 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:09:50 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 18:09:50 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 18:09:50 2021 - [info] GTID ON
Wed May 19 18:09:50 2021 - [info] Replicating from 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:09:50 2021 - [info] Not candidate for the new Master (no_master is set)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.55.106(192.168.55.106:5566)? (YES/no): YES
Wed May 19 18:09:53 2021 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Wed May 19 18:09:53 2021 - [info] ok.
Wed May 19 18:09:53 2021 - [info] Checking MHA is not monitoring or doing failover..
Wed May 19 18:09:53 2021 - [info] Checking replication health on 192.168.55.105..
Wed May 19 18:09:53 2021 - [info] ok.
Wed May 19 18:09:53 2021 - [info] Checking replication health on 192.168.55.108..
Wed May 19 18:09:53 2021 - [info] ok.
Wed May 19 18:09:53 2021 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln1218] 192.168.55.105 is not alive!
Wed May 19 18:09:53 2021 - [error][/usr/local/share/perl5/MHA/MasterRotate.pm, ln233] Failed to get new master!
Wed May 19 18:09:53 2021 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/local/bin/masterha_master_switch line 53.

 

由于设置过candidate_master，经过测试在手动切换的时候不能再指定--new_master_host，看来不能画蛇添足！！！
[root@host-192-168-55-103 app1]#
[root@host-192-168-55-103 app1]# masterha_master_switch --master_state=alive --conf=/etc/masterha/app1/app1.conf --orig_master_is_new_slave
Wed May 19 18:21:50 2021 - [info] MHA::MasterRotate version 0.58.
Wed May 19 18:21:50 2021 - [info] Starting online master switch..
Wed May 19 18:21:50 2021 - [info]
Wed May 19 18:21:50 2021 - [info] * Phase 1: Configuration Check Phase..
Wed May 19 18:21:50 2021 - [info]
Wed May 19 18:21:50 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed May 19 18:21:50 2021 - [info] Reading application default configuration from /etc/masterha/app1/app1.conf..
Wed May 19 18:21:50 2021 - [info] Reading server configuration from /etc/masterha/app1/app1.conf..
Wed May 19 18:21:51 2021 - [info] GTID failover mode = 1
Wed May 19 18:21:51 2021 - [info] Current Alive Master: 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:21:51 2021 - [info] Alive Slaves:
Wed May 19 18:21:51 2021 - [info] 192.168.55.105(192.168.55.105:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 18:21:51 2021 - [info] GTID ON
Wed May 19 18:21:51 2021 - [info] Replicating from 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:21:51 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 18:21:51 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 18:21:51 2021 - [info] GTID ON
Wed May 19 18:21:51 2021 - [info] Replicating from 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:21:51 2021 - [info] Not candidate for the new Master (no_master is set)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.55.106(192.168.55.106:5566)? (YES/no): YES
Wed May 19 18:21:56 2021 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Wed May 19 18:21:56 2021 - [info] ok.
Wed May 19 18:21:56 2021 - [info] Checking MHA is not monitoring or doing failover..
Wed May 19 18:21:56 2021 - [info] Checking replication health on 192.168.55.105..
Wed May 19 18:21:56 2021 - [info] ok.
Wed May 19 18:21:56 2021 - [info] Checking replication health on 192.168.55.108..
Wed May 19 18:21:56 2021 - [info] ok.
Wed May 19 18:21:56 2021 - [info] Searching new master from slaves..
Wed May 19 18:21:56 2021 - [info] Candidate masters from the configuration file:
Wed May 19 18:21:56 2021 - [info] 192.168.55.105(192.168.55.105:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 18:21:56 2021 - [info] GTID ON
Wed May 19 18:21:56 2021 - [info] Replicating from 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:21:56 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Wed May 19 18:21:56 2021 - [info] 192.168.55.106(192.168.55.106:5566) Version=5.7.24-27-log log-bin:enabled
Wed May 19 18:21:56 2021 - [info] GTID ON
Wed May 19 18:21:56 2021 - [info] Non-candidate masters:
Wed May 19 18:21:56 2021 - [info] 192.168.55.108(192.168.55.108:5566) Version=5.7.24-27-log (oldest major version between slaves) log-bin:enabled
Wed May 19 18:21:56 2021 - [info] GTID ON
Wed May 19 18:21:56 2021 - [info] Replicating from 192.168.55.106(192.168.55.106:5566)
Wed May 19 18:21:56 2021 - [info] Not candidate for the new Master (no_master is set)
Wed May 19 18:21:56 2021 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Wed May 19 18:21:56 2021 - [info]
From:
192.168.55.106(192.168.55.106:5566) (current master)
+--192.168.55.105(192.168.55.105:5566)
+--192.168.55.108(192.168.55.108:5566)

To:
192.168.55.105(192.168.55.105:5566) (new master)
+--192.168.55.108(192.168.55.108:5566)
+--192.168.55.106(192.168.55.106:5566)

Starting master switch from 192.168.55.106(192.168.55.106:5566) to 192.168.55.105(192.168.55.105:5566)? (yes/NO): yes
Wed May 19 18:22:18 2021 - [info] Checking whether 192.168.55.105(192.168.55.105:5566) is ok for the new master..
Wed May 19 18:22:18 2021 - [info] ok.
Wed May 19 18:22:18 2021 - [info] 192.168.55.106(192.168.55.106:5566): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Wed May 19 18:22:18 2021 - [info] 192.168.55.106(192.168.55.106:5566): Resetting slave pointing to the dummy host.
Wed May 19 18:22:18 2021 - [info] ** Phase 1: Configuration Check Phase completed.
Wed May 19 18:22:18 2021 - [info]
Wed May 19 18:22:18 2021 - [info] * Phase 2: Rejecting updates Phase..
Wed May 19 18:22:18 2021 - [info]
Wed May 19 18:22:18 2021 - [info] Executing master ip online change script to disable write on the current master:
Wed May 19 18:22:18 2021 - [info] /usr/local/scripts/master_ip_online_change --command=stop --orig_master_host=192.168.55.106 --orig_master_ip=192.168.55.106 --orig_master_port=5566 --orig_master_user='root' --new_master_host=192.168.55.105 --new_master_ip=192.168.55.105 --new_master_port=5566 --new_master_user='root' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Wed May 19 18:22:18 2021 584498 Set read_only on the new master.. ok.
Wed May 19 18:22:18 2021 586620 drop vip 192.168.55.200..
RTNETLINK answers: Cannot assign requested address
Wed May 19 18:22:18 2021 680789 Waiting all running 2 threads are disconnected.. (max 1500 milliseconds)
{'Time' => '4666','db' => undef,'Id' => '37','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.108:45825'}
{'Time' => '3878','db' => undef,'Id' => '43','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.105:56062'}
Wed May 19 18:22:19 2021 181253 Waiting all running 2 threads are disconnected.. (max 1000 milliseconds)
{'Time' => '4666','db' => undef,'Id' => '37','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.108:45825'}
{'Time' => '3878','db' => undef,'Id' => '43','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.105:56062'}
Wed May 19 18:22:19 2021 681814 Waiting all running 2 threads are disconnected.. (max 500 milliseconds)
{'Time' => '4667','db' => undef,'Id' => '37','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.108:45825'}
{'Time' => '3879','db' => undef,'Id' => '43','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.105:56062'}
Wed May 19 18:22:20 2021 182759 Set read_only=1 on the orig master.. ok.
Wed May 19 18:22:20 2021 196758 Waiting all running 2 queries are disconnected.. (max 500 milliseconds)
{'Time' => '4667','db' => undef,'Id' => '37','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.108:45825'}
{'Time' => '3879','db' => undef,'Id' => '43','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Rows_examined' => '0','Info' => undef,'Rows_sent' => '0','Host' => '192.168.55.105:56062'}
Wed May 19 18:22:20 2021 683089 Killing all application threads..
Wed May 19 18:22:20 2021 684547 done.
Wed May 19 18:22:20 2021 - [info] ok.
Wed May 19 18:22:20 2021 - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Wed May 19 18:22:20 2021 - [info] Executing FLUSH TABLES WITH READ LOCK..
Wed May 19 18:22:20 2021 - [info] ok.
Wed May 19 18:22:20 2021 - [info] Orig master binlog:pos is mysql-bin.000002:1948.
Wed May 19 18:22:20 2021 - [info] Waiting to execute all relay logs on 192.168.55.105(192.168.55.105:5566)..
Wed May 19 18:22:20 2021 - [info] master_pos_wait(mysql-bin.000002:1948) completed on 192.168.55.105(192.168.55.105:5566). Executed 0 events.
Wed May 19 18:22:20 2021 - [info] done.
Wed May 19 18:22:20 2021 - [info] Getting new master's binlog name and position..
Wed May 19 18:22:20 2021 - [info] mysql-bin.000003:986
Wed May 19 18:22:20 2021 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.55.105', MASTER_PORT=5566, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Wed May 19 18:22:20 2021 - [info] Executing master ip online change script to allow write on the new master:
Wed May 19 18:22:20 2021 - [info] /usr/local/scripts/master_ip_online_change --command=start --orig_master_host=192.168.55.106 --orig_master_ip=192.168.55.106 --orig_master_port=5566 --orig_master_user='root' --new_master_host=192.168.55.105 --new_master_ip=192.168.55.105 --new_master_port=5566 --new_master_user='root' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Wed May 19 18:22:20 2021 908398 Set read_only=0 on the new master.
Wed May 19 18:22:20 2021 908916Add vip 192.168.55.200 on eth0..
Wed May 19 18:22:21 2021 - [info] ok.
Wed May 19 18:22:21 2021 - [info]
Wed May 19 18:22:21 2021 - [info] * Switching slaves in parallel..
Wed May 19 18:22:21 2021 - [info]
Wed May 19 18:22:21 2021 - [info] -- Slave switch on host 192.168.55.108(192.168.55.108:5566) started, pid: 6210
Wed May 19 18:22:21 2021 - [info]
Wed May 19 18:22:22 2021 - [info] Log messages from 192.168.55.108 ...
Wed May 19 18:22:22 2021 - [info]
Wed May 19 18:22:21 2021 - [info] Waiting to execute all relay logs on 192.168.55.108(192.168.55.108:5566)..
Wed May 19 18:22:21 2021 - [info] master_pos_wait(mysql-bin.000002:1948) completed on 192.168.55.108(192.168.55.108:5566). Executed 0 events.
Wed May 19 18:22:21 2021 - [info] done.
Wed May 19 18:22:21 2021 - [info] Resetting slave 192.168.55.108(192.168.55.108:5566) and starting replication from the new master 192.168.55.105(192.168.55.105:5566)..
Wed May 19 18:22:21 2021 - [info] Executed CHANGE MASTER.
Wed May 19 18:22:21 2021 - [info] Slave started.
Wed May 19 18:22:22 2021 - [info] End of log messages from 192.168.55.108 ...
Wed May 19 18:22:22 2021 - [info]
Wed May 19 18:22:22 2021 - [info] -- Slave switch on host 192.168.55.108(192.168.55.108:5566) succeeded.
Wed May 19 18:22:22 2021 - [info] Unlocking all tables on the orig master:
Wed May 19 18:22:22 2021 - [info] Executing UNLOCK TABLES..
Wed May 19 18:22:22 2021 - [info] ok.
Wed May 19 18:22:22 2021 - [info] Starting orig master as a new slave..
Wed May 19 18:22:22 2021 - [info] Resetting slave 192.168.55.106(192.168.55.106:5566) and starting replication from the new master 192.168.55.105(192.168.55.105:5566)..
Wed May 19 18:22:22 2021 - [info] Executed CHANGE MASTER.
Wed May 19 18:22:22 2021 - [info] Slave started.
Wed May 19 18:22:22 2021 - [info] All new slave servers switched successfully.
Wed May 19 18:22:22 2021 - [info]
Wed May 19 18:22:22 2021 - [info] * Phase 5: New master cleanup phase..
Wed May 19 18:22:22 2021 - [info]
Wed May 19 18:22:22 2021 - [info] 192.168.55.105: Resetting slave info succeeded.
Wed May 19 18:22:22 2021 - [info] Switching master to 192.168.55.105(192.168.55.105:5566) completed successfully.
[root@host-192-168-55-103 app1]#

 

 

/usr/local/send_report脚本示例：

mha_log="/var/log/masterha/app1/manager.log"
num=$(grep -n -w 'Failover Report' $mha_log | tail -n1 | awk -F':' '{print $1}')
check_time=$(date +"%F-%T")
let start_num=num+2
let end_num=num+2
msg_info="$(sed -n "${start_num},${end_num}p" $mha_log)"
old_master=$(echo "$msg_info"  | awk '{print $5}' | awk -F'[(|)]' '{print $2}')
new_master=$(echo "$msg_info"  | awk '{print $7}' | awk -F'[(|)]' '{print $2}')

curl 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxx' \
   -H 'Content-Type: application/json' \
   -d '
   {
        "msgtype": "text",
        "text": {
            "content": "【'$check_time' app1应用数据库failover】\nfrom\n'$old_master'\nto\n'$new_master'",
       	    "mentioned_mobile_list":["电话号码"]
        }
   }'

　　

failover切换测试和告警

 

 

posted @ 2021-05-26 17:32  knowledge-is-power  阅读(440)  评论(0)  编辑  收藏  举报