MHA切换检测日志分析
GTID模式
[root@db03 ~]
Tue Nov 19 17:52:00 2019 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Tue Nov 19 17:52:00 2019 - [info] Executing SSH check script: exit 0
Tue Nov 19 17:52:00 2019 - [info] HealthCheck: SSH to 10.0.0.51 is reachable.
Tue Nov 19 17:52:02 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.0.0.51' (111))
#连接失败2次。。
Tue Nov 19 17:52:02 2019 - [warning] Connection failed 2 time(s)..
#在MySQL connect:2003上出现错误(无法连接到“10.0.0.51”(111)上的MySQL服务器)
Tue Nov 19 17:52:04 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.0.0.51' (111))
Tue Nov 19 17:52:04 2019 - [warning] Connection failed 3 time(s)..
Tue Nov 19 17:52:06 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.0.0.51' (111))
#连接失败4次。。
Tue Nov 19 17:52:06 2019 - [warning] Connection failed 4 time(s)..
#无法从健康检查程序访问主服务器!
Tue Nov 19 17:52:06 2019 - [warning] Master is not reachable from health checker!
#无法访问Master 10.0.0.51(10.0.0.51:3306)!
Tue Nov 19 17:52:06 2019 - [warning] Master 10.0.0.51(10.0.0.51:3306) is not reachable!
#SSH是可访问的。
Tue Nov 19 17:52:06 2019 - [warning] SSH is reachable.
#连接到主服务器失败。再次读取配置文件/etc/masterha_default.cnf和/etc/mha/app1.cnf,并尝试连接到所有服务器以检查服务器状态。
Tue Nov 19 17:52:06 2019 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1.cnf again, and trying to connect to all servers to check server status..
#未找到全局配置文件/etc/masterha/u default.cnf。跳过!
Tue Nov 19 17:52:06 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
#正在从/etc/mha/app1.cnf.读取应用程序默认配置。。
Tue Nov 19 17:52:06 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
# 正在从/etc/mha/app1.cnf.读取服务器配置。。
Tue Nov 19 17:52:06 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
#GTID故障转移模式=1
Tue Nov 19 17:52:07 2019 - [info] GTID failover mode = 1
#死机服务器:
Tue Nov 19 17:52:07 2019 - [info] Dead Servers:
#10.0.0.51(10.0.0.51:3306)
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.51(10.0.0.51:3306)
#活动服务器:
Tue Nov 19 17:52:07 2019 - [info] Alive Servers:
#10.0.0.52(10.0.0.52:3306)
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.52(10.0.0.52:3306)
#10.0.0.54(10.0.0.54:3306)
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.54(10.0.0.54:3306)
#10.0.0.53(10.0.0.53:3306)
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.53(10.0.0.53:3306)
#活着的奴隶:
Tue Nov 19 17:52:07 2019 - [info] Alive Slaves:
#10.0.0.52(10.0.0.52:3306)版本=5.6.44-log(从机之间的主要旧版本)日志箱:
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:07 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:07 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54(10.0.0.54:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:07 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:07 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53(10.0.0.53:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:07 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:07 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:07 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#检查从机配置。。
Tue Nov 19 17:52:07 2019 - [info] Checking slave configurations..
#只读=1未设置在从机10.0.0.52(10.0.0.52:3306)上。
Tue Nov 19 17:52:07 2019 - [info] read_only=1 is not set on slave 10.0.0.52(10.0.0.52:3306).
#只读=1未设置在从机10.0.0.54(10.0.0.54:3306)上。
Tue Nov 19 17:52:07 2019 - [info] read_only=1 is not set on slave 10.0.0.54(10.0.0.54:3306).
#只读=1未设置在从机10.0.0.53(10.0.0.53:3306)上。
Tue Nov 19 17:52:07 2019 - [info] read_only=1 is not set on slave 10.0.0.53(10.0.0.53:3306).
#正在检查复制筛选设置。。
Tue Nov 19 17:52:07 2019 - [info] Checking replication filtering settings..
#复制筛选检查正常。
Tue Nov 19 17:52:07 2019 - [info] Replication filtering check ok.
#主人倒下了!
Tue Nov 19 17:52:07 2019 - [info] Master is down!
#终止监视脚本。
Tue Nov 19 17:52:07 2019 - [info] Terminating monitoring script.
#获得退出代码20(主死亡)。
Tue Nov 19 17:52:07 2019 - [info] Got exit code 20 (Master dead).
#MHA::主故障转移版本0.56。
Tue Nov 19 17:52:07 2019 - [info] MHA::MasterFailover version 0.56.
#启动主故障转移。
Tue Nov 19 17:52:07 2019 - [info] Starting master failover.
Tue Nov 19 17:52:07 2019 - [info]
#*第1阶段:配置检查阶段。。
Tue Nov 19 17:52:07 2019 - [info] * Phase 1: Configuration Check Phase..
Tue Nov 19 17:52:07 2019 - [info]
#GTID故障转移模式=1
Tue Nov 19 17:52:08 2019 - [info] GTID failover mode = 1
#死机服务器:
Tue Nov 19 17:52:08 2019 - [info] Dead Servers:
#10.0.0.51(10.0.0.51:3306)
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.51(10.0.0.51:3306)
#通过MySQL检查主机可访问性(双重检查…)。。。
Tue Nov 19 17:52:08 2019 - [info] Checking master reachability via MySQL(double check)...
#好。
Tue Nov 19 17:52:08 2019 - [info] ok.
#活动服务器:
Tue Nov 19 17:52:08 2019 - [info] Alive Servers:
#10.0.0.52(10.0.0.52:3306)
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.52(10.0.0.52:3306)
#10.0.0.54(10.0.0.54:3306)
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.54(10.0.0.54:3306)
#10.0.0.53(10.0.0.53:3306)
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.53(10.0.0.53:3306)
#活奴隶:
Tue Nov 19 17:52:08 2019 - [info] Alive Slaves:
#10.0.0.52(10.0.0.52:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:08 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:08 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54(10.0.0.54:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:08 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:08 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53(10.0.0.53:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:08 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:08 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:08 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#启动基于GTID的故障转移。
Tue Nov 19 17:52:08 2019 - [info] Starting GTID based failover.
Tue Nov 19 17:52:08 2019 - [info]
#**第1阶段:配置检查阶段完成。
Tue Nov 19 17:52:08 2019 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Nov 19 17:52:08 2019 - [info]
#*第2阶段:主停堆阶段。。
Tue Nov 19 17:52:08 2019 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Nov 19 17:52:08 2019 - [info]
#强制关闭,使应用程序永远无法连接到当前主服务器。。
Tue Nov 19 17:52:08 2019 - [info] Forcing shutdown so that applications never connect to the current master..
#[警告]未设置master_ip_failover_脚本。正在跳过失效的主IP地址
Tue Nov 19 17:52:08 2019 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
#[警告]关闭脚本未设置。跳过显式关闭已死亡的主控形状。
Tue Nov 19 17:52:08 2019 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
#*第2阶段:主停堆阶段完成。
Tue Nov 19 17:52:09 2019 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Nov 19 17:52:09 2019 - [info]
#*第3阶段:主恢复阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3: Master Recovery Phase..
Tue Nov 19 17:52:09 2019 - [info]
#*第3.1阶段:获取最新的从属阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Nov 19 17:52:09 2019 - [info]
#所有从属服务器上的最新二进制日志文件/位置是mysql bin.000005:191
Tue Nov 19 17:52:09 2019 - [info] The latest binary log file/position on all slaves is mysql-bin.000005:191
#检索到Gtid集:6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
Tue Nov 19 17:52:09 2019 - [info] Retrieved Gtid Set: 6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
#最新从属服务器(接收到最新中继日志文件的从属服务器):
Tue Nov 19 17:52:09 2019 - [info] Latest slaves (Slaves that received relay log files to the latest):
#10.0.0.52(10.0.0.52:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54(10.0.0.54:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53(10.0.0.53:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#所有从机上最早的二进制日志文件/位置是mysql bin.000005:191
Tue Nov 19 17:52:09 2019 - [info] The oldest binary log file/position on all slaves is mysql-bin.000005:191
#检索到Gtid集:6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
Tue Nov 19 17:52:09 2019 - [info] Retrieved Gtid Set: 6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
#最老的奴隶:
Tue Nov 19 17:52:09 2019 - [info] Oldest slaves:
#10.0.0.52(10.0.0.52:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.54(10.0.0.54:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.54(10.0.0.54:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
#10.0.0.53(10.0.0.53:3306)版本=5.6.44-log(从机之间最旧的主版本)日志箱:启用
Tue Nov 19 17:52:09 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.6.44-log (oldest major version between slaves) log-bin:enabled
#GTID开启
Tue Nov 19 17:52:09 2019 - [info] GTID ON
#从10.0.0.51复制(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
Tue Nov 19 17:52:09 2019 - [info]
#*第3.3阶段:确定新的主阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3.3: Determining New Master Phase..
Tue Nov 19 17:52:09 2019 - [info]
#从奴隶中搜索新的主。。
Tue Nov 19 17:52:09 2019 - [info] Searching new master from slaves..
#配置文件中的候选主控形状:
Tue Nov 19 17:52:09 2019 - [info] Candidate masters from the configuration file:
#非候选硕士:
Tue Nov 19 17:52:09 2019 - [info] Non-candidate masters:
#新主机为10.0.0.52(10.0.0.52:3306)
Tue Nov 19 17:52:09 2019 - [info] New master is 10.0.0.52(10.0.0.52:3306)
#启动主故障转移。。
Tue Nov 19 17:52:09 2019 - [info] Starting master failover..
Tue Nov 19 17:52:09 2019 - [info]
#发件人:
From:
#10.0.0.51(10.0.0.51:3306)(当前主控)
10.0.0.51(10.0.0.51:3306) (current master)
+--10.0.0.52(10.0.0.52:3306)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
#致:
To:
#10.0.0.52(10.0.0.52:3306)(新母版)
10.0.0.52(10.0.0.52:3306) (new master)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
Tue Nov 19 17:52:09 2019 - [info]
#*第3.3阶段:新主恢复阶段。。
Tue Nov 19 17:52:09 2019 - [info] * Phase 3.3: New Master Recovery Phase..
Tue Nov 19 17:52:09 2019 - [info]
#等待应用所有日志。。
Tue Nov 19 17:52:09 2019 - [info] Waiting all logs to be applied..
#完成。
Tue Nov 19 17:52:09 2019 - [info] done.
#获取新主人的binlog名称和位置。。
Tue Nov 19 17:52:09 2019 - [info] Getting new master's binlog name and position..
Tue Nov 19 17:52:09 2019 - [info] mysql-bin.000002:438
Tue Nov 19 17:52:09 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='slave', MASTER_PASSWORD='xxx';
Tue Nov 19 17:52:09 2019 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000002, 438, 6e607afa-0a67-11ea-ac08-000c29f3321a:1-2
Tue Nov 19 17:52:09 2019 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Tue Nov 19 17:52:09 2019 - [info] ** Finished master recovery successfully.
Tue Nov 19 17:52:09 2019 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov 19 17:52:09 2019 - [info]
Tue Nov 19 17:52:09 2019 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov 19 17:52:09 2019 - [info]
Tue Nov 19 17:52:09 2019 - [info]
Tue Nov 19 17:52:09 2019 - [info] * Phase 4.1: Starting Slaves in parallel..
Tue Nov 19 17:52:09 2019 - [info]
Tue Nov 19 17:52:09 2019 - [info] -- Slave recovery on host 10.0.0.54(10.0.0.54:3306) started, pid: 22464. Check tmp log /etc/mha/app1/10.0.0.54_3306_20191119175207.log if it takes time..
Tue Nov 19 17:52:09 2019 - [info] -- Slave recovery on host 10.0.0.53(10.0.0.53:3306) started, pid: 22465. Check tmp log /etc/mha/app1/10.0.0.53_3306_20191119175207.log if it takes time..
Tue Nov 19 17:52:10 2019 - [info]
Tue Nov 19 17:52:10 2019 - [info] Log messages from 10.0.0.54 ...
Tue Nov 19 17:52:10 2019 - [info]
Tue Nov 19 17:52:09 2019 - [info] Resetting slave
10.0.0.54(10.0.0.54:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
Tue Nov 19 17:52:09 2019 - [info] Executed CHANGE MASTER.
Tue Nov 19 17:52:09 2019 - [info] Slave started.
Tue Nov 19 17:52:09 2019 - [info] gtid_wait(6e607afa-0a67-11ea-ac08-000c29f3321a:1-2) completed on 10.0.0.54(10.0.0.54:3306). Executed 0 events.
Tue Nov 19 17:52:10 2019 - [info] End of log messages from 10.0.0.54.
Tue Nov 19 17:52:10 2019 - [info] -- Slave on host 10.0.0.54(10.0.0.54:3306) started.
Tue Nov 19 17:52:11 2019 - [info]
Tue Nov 19 17:52:11 2019 - [info] Log messages from 10.0.0.53 ...
Tue Nov 19 17:52:11 2019 - [info]
Tue Nov 19 17:52:09 2019 - [info] Resetting slave
10.0.0.53(10.0.0.53:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
Tue Nov 19 17:52:09 2019 - [info] Executed CHANGE MASTER.
Tue Nov 19 17:52:10 2019 - [info] Slave started.
Tue Nov 19 17:52:10 2019 - [info] gtid_wait(6e607afa-0a67-11ea-ac08-000c29f3321a:1-2) completed on 10.0.0.53(10.0.0.53:3306). Executed 0 events.
Tue Nov 19 17:52:11 2019 - [info] End of log messages from 10.0.0.53.
Tue Nov 19 17:52:11 2019 - [info] -- Slave on host 10.0.0.53(10.0.0.53:3306) started.
Tue Nov 19 17:52:11 2019 - [info] All new slave servers recovered successfully.
Tue Nov 19 17:52:11 2019 - [info]
Tue Nov 19 17:52:11 2019 - [info] * Phase 5: New master cleanup phase..
Tue Nov 19 17:52:11 2019 - [info]
Tue Nov 19 17:52:11 2019 - [info] Resetting slave info on the new master..
Tue Nov 19 17:52:11 2019 - [info] 10.0.0.52: Resetting slave info succeeded.
Tue Nov 19 17:52:11 2019 - [info] Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.
Tue Nov 19 17:52:11 2019 - [info] Deleted server1 entry from /etc/mha/app1.cnf .
Tue Nov 19 17:52:11 2019 - [info]
----- Failover Report -----
app1: MySQL Master failover 10.0.0.51(10.0.0.51:3306) to 10.0.0.52(10.0.0.52:3306) succeeded
Master 10.0.0.51(10.0.0.51:3306) is down!
Check MHA Manager logs at db03:/etc/mha/manager.log for details.
Started automated(non-interactive) failover.
Selected 10.0.0.52(10.0.0.52:3306) as a new master.
10.0.0.52(10.0.0.52:3306): OK: Applying all logs succeeded.
10.0.0.54(10.0.0.54:3306): OK: Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
10.0.0.53(10.0.0.53:3306): OK: Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
10.0.0.52(10.0.0.52:3306): Resetting slave info succeeded.
Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.
普通模式
From:
10.0.0.51(10.0.0.51:3306) (current master)
+--10.0.0.52(10.0.0.52:3306)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
To:
10.0.0.52(10.0.0.52:3306) (new master)
+--10.0.0.54(10.0.0.54:3306)
+--10.0.0.53(10.0.0.53:3306)
Tue Nov 19 19:22:21 2019 - [info]
Tue Nov 19 19:22:21 2019 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Tue Nov 19 19:22:21 2019 - [info]
Tue Nov 19 19:22:21 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Nov 19 19:22:21 2019 - [info] Sending binlog..
Tue Nov 19 19:22:22 2019 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog to root@10.0.0.52:/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog succeeded.
Tue Nov 19 19:22:22 2019 - [info]
Tue Nov 19 19:22:22 2019 - [info] * Phase 3.4: Master Log Apply Phase..
Tue Nov 19 19:22:22 2019 - [info]
Tue Nov 19 19:22:22 2019 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Nov 19 19:22:22 2019 - [info] Starting recovery on 10.0.0.52(10.0.0.52:3306)..
Tue Nov 19 19:22:22 2019 - [info] Generating diffs succeeded.
Tue Nov 19 19:22:22 2019 - [info] Waiting until all relay logs are applied.
Tue Nov 19 19:22:22 2019 - [info] done.
Tue Nov 19 19:22:22 2019 - [info] Getting slave status..
Tue Nov 19 19:22:22 2019 - [info] This slave(10.0.0.52)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
#连接到目标从属主机10.0.0.52,运行恢复脚本。。
Tue Nov 19 19:22:22 2019 - [info] Connecting to the target slave host 10.0.0.52, running recover script..
#执行命令:
Tue Nov 19 19:22:22 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.0.0.52 --slave_ip=10.0.0.52 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog --workdir=/var/tmp --target_version=5.6.44-log --timestamp=20191119192216 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Nov 19 19:22:22 2019 - [info]
#MySQL客户端版本是5.6.44。使用--二进制模式。应用差分二进制/中继日志文件/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.52:3306。这可能需要很长时间。。。
MySQL client version is 5.6.44. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.52:3306. This may take long time...
#应用日志文件成功。
Applying log files succeeded.
#所有中继日志均已成功应用。
Tue Nov 19 19:22:22 2019 - [info] All relay logs were successfully applied.
#获取新主人的binlog名称和位置。。
Tue Nov 19 19:22:22 2019 - [info] Getting new master's binlog name and position..
Tue Nov 19 19:22:22 2019 - [info] mysql-bin.000003:120
Tue Nov 19 19:22:22 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=120, MASTER_USER='slave', MASTER_PASSWORD='xxx';
Tue Nov 19 19:22:22 2019 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Tue Nov 19 19:22:22 2019 - [info] ** Finished master recovery successfully.
Tue Nov 19 19:22:22 2019 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov 19 19:22:22 2019 - [info]
Tue Nov 19 19:22:22 2019 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov 19 19:22:22 2019 - [info]
Tue Nov 19 19:22:22 2019 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Nov 19 19:22:22 2019 - [info]
Tue Nov 19 19:22:22 2019 - [info] -- Slave diff file generation on host 10.0.0.54(10.0.0.54:3306) started, pid: 23478. Check tmp log /etc/mha/app1/10.0.0.54_3306_20191119192216.log if it takes time..
Tue Nov 19 19:22:22 2019 - [info] -- Slave diff file generation on host 10.0.0.53(10.0.0.53:3306) started, pid: 23479. Check tmp log
/etc/mha/app1/10.0.0.53_3306_20191119192216.log if it takes time..
Tue Nov 19 19:22:23 2019 - [info]
Tue Nov 19 19:22:23 2019 - [info] Log messages from 10.0.0.54 ...
Tue Nov 19 19:22:23 2019 - [info]
Tue Nov 19 19:22:22 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Nov 19 19:22:23 2019 - [info] End of log messages from 10.0.0.54.
Tue Nov 19 19:22:23 2019 - [info] -- 10.0.0.54(10.0.0.54:3306) has the latest relay log events.
Tue Nov 19 19:22:23 2019 - [info]
Tue Nov 19 19:22:23 2019 - [info] Log messages from 10.0.0.53 ...
Tue Nov 19 19:22:23 2019 - [info]
Tue Nov 19 19:22:22 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Nov 19 19:22:23 2019 - [info] End of log messages from 10.0.0.53.
Tue Nov 19 19:22:23 2019 - [info] -- 10.0.0.53(10.0.0.53:3306) has the latest relay log events.
Tue Nov 19 19:22:23 2019 - [info] Generating relay diff files from the latest slave succeeded.
Tue Nov 19 19:22:23 2019 - [info]
Tue Nov 19 19:22:23 2019 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Nov 19 19:22:23 2019 - [info]
Tue Nov 19 19:22:23 2019 - [info] -- Slave recovery on host 10.0.0.54(10.0.0.54:3306) started, pid: 23482. Check tmp log /etc/mha/app1/10.0.0.54_3306_20191119192216.log if it takes time..
Tue Nov 19 19:22:23 2019 - [info] -- Slave recovery on host 10.0.0.53(10.0.0.53:3306) started, pid: 23483. Check tmp log /etc/mha/app1/10.0.0.53_3306_20191119192216.log if it takes time..
Tue Nov 19 19:22:24 2019 - [info]
Tue Nov 19 19:22:24 2019 - [info] Log messages from 10.0.0.54 ...
Tue Nov 19 19:22:24 2019 - [info]
Tue Nov 19 19:22:23 2019 - [info] Sending binlog..
Tue Nov 19 19:22:23 2019 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog to root@10.0.0.54:/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog succeeded.
Tue Nov 19 19:22:23 2019 - [info] Starting recovery on 10.0.0.54(10.0.0.54:3306)..
Tue Nov 19 19:22:23 2019 - [info] Generating diffs succeeded.
Tue Nov 19 19:22:23 2019 - [info] Waiting until all relay logs are applied.
Tue Nov 19 19:22:23 2019 - [info] done.
Tue Nov 19 19:22:23 2019 - [info] Getting slave status..
Tue Nov 19 19:22:23 2019 - [info] This slave(10.0.0.54)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
#连接到目标从属主机10.0.0.54,运行恢复脚本。。
Tue Nov 19 19:22:23 2019 - [info] Connecting to the target slave host 10.0.0.54, running recover script..
#执行命令:
Tue Nov 19 19:22:23 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.0.0.54 --slave_ip=10.0.0.54 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog --workdir=/var/tmp --target_version=5.6.44-log --timestamp=20191119192216 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Nov 19 19:22:24 2019 - [info]
#MySQL客户端版本是5.6.44。使用--二进制模式。应用差分二进制/中继日志文件/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.54:3306。这可能需要很长时间。。。 应用日志文件成功。
MySQL client version is 5.6.44. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.54:3306. This may take long time...
Applying log files succeeded.
#所有中继日志均已成功应用。
Tue Nov 19 19:22:24 2019 - [info] All relay logs were successfully applied.
#重置从机10.0.0.54(10.0.0.54:3306)并从新主机10.0.0.52(10.0.0.52:3306)开始复制。。
Tue Nov 19 19:22:24 2019 - [info] Resetting slave 10.0.0.54(10.0.0.54:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
#执行变更主控。
Tue Nov 19 19:22:24 2019 - [info] Executed CHANGE MASTER.
#奴隶启动。
Tue Nov 19 19:22:24 2019 - [info] Slave started.
#10.0.0.54的日志消息结束。
Tue Nov 19 19:22:24 2019 - [info] End of log messages from 10.0.0.54.
#——主机10.0.0.54(10.0.0.54:3306)上的从机恢复成功。
Tue Nov 19 19:22:24 2019 - [info] -- Slave recovery on host 10.0.0.54(10.0.0.54:3306) succeeded.
Tue Nov 19 19:22:25 2019 - [info]
#记录来自10.0.0.53的消息。。。
Tue Nov 19 19:22:25 2019 - [info] Log messages from 10.0.0.53 ...
Tue Nov 19 19:22:25 2019 - [info]
#发送binlog。。
Tue Nov 19 19:22:23 2019 - [info] Sending binlog..
Tue Nov 19 19:22:24 2019 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog to root@10.0.0.53:/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog succeeded.
#10.0.0.53(10.0.0.53:3306)开始恢复。。
Tue Nov 19 19:22:24 2019 - [info] Starting recovery on 10.0.0.53(10.0.0.53:3306)..
#生成差异成功。
Tue Nov 19 19:22:24 2019 - [info] Generating diffs succeeded.
#等待所有中继日志应用。
Tue Nov 19 19:22:24 2019 - [info] Waiting until all relay logs are applied.
#完成。
Tue Nov 19 19:22:24 2019 - [info] done.
#获取奴隶状态。。
Tue Nov 19 19:22:24 2019 - [info] Getting slave status..
#此从机(10.0.0.53)的Exec_Master_Log_Pos等于Read_Master_Log_Pos(mysql bin.000006:120)。无需从Exec_Master_Log_Pos恢复。
Tue Nov 19 19:22:24 2019 - [info] This slave(10.0.0.53)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
Tue Nov 19 19:22:24 2019 - [info] Connecting to the target slave host 10.0.0.53, running recover script..
Tue Nov 19 19:22:24 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.0.0.53 --slave_ip=10.0.0.53 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog --workdir=/var/tmp --target_version=5.6.44-log --timestamp=20191119192216 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Nov 19 19:22:24 2019 - [info]
MySQL client version is 5.6.44. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_10.0.0.51_3306_20191119192216.binlog on 10.0.0.53:3306. This may take long time...
Applying log files succeeded.
Tue Nov 19 19:22:24 2019 - [info] All relay logs were successfully applied.
Tue Nov 19 19:22:24 2019 - [info] Resetting slave 10.0.0.53(10.0.0.53:3306) and starting replication from the new master 10.0.0.52(10.0.0.52:3306)..
Tue Nov 19 19:22:24 2019 - [info] Executed CHANGE MASTER.
Tue Nov 19 19:22:24 2019 - [info] Slave started.
Tue Nov 19 19:22:25 2019 - [info] End of log messages from 10.0.0.53.
Tue Nov 19 19:22:25 2019 - [info] -- Slave recovery on host 10.0.0.53(10.0.0.53:3306) succeeded.
Tue Nov 19 19:22:25 2019 - [info] All new slave servers recovered successfully.
Tue Nov 19 19:22:25 2019 - [info]
Tue Nov 19 19:22:25 2019 - [info] * Phase 5: New master cleanup phase..
Tue Nov 19 19:22:25 2019 - [info]
Tue Nov 19 19:22:25 2019 - [info] Resetting slave info on the new master..
Tue Nov 19 19:22:25 2019 - [info] 10.0.0.52: Resetting slave info succeeded.
Tue Nov 19 19:22:25 2019 - [info] Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.
Tue Nov 19 19:22:25 2019 - [info] Deleted server1 entry from /etc/mha/app1.cnf .
Tue Nov 19 19:22:25 2019 - [info]
----- Failover Report -----
app1: MySQL Master failover 10.0.0.51(10.0.0.51:3306) to 10.0.0.52(10.0.0.52:3306) succeeded
Master 10.0.0.51(10.0.0.51:3306) is down!
Check MHA Manager logs at db03:/etc/mha/manager.log for details.
Started automated(non-interactive) failover.
The latest slave 10.0.0.52(10.0.0.52:3306) has all relay logs for recovery.
Selected 10.0.0.52(10.0.0.52:3306) as a new master.
10.0.0.52(10.0.0.52:3306): OK: Applying all logs succeeded.
10.0.0.54(10.0.0.54:3306): This host has the latest relay log events.
10.0.0.53(10.0.0.53:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
10.0.0.54(10.0.0.54:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
10.0.0.53(10.0.0.53:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.0.0.52(10.0.0.52:3306)
10.0.0.52(10.0.0.52:3306): Resetting slave info succeeded.
Master failover to 10.0.0.52(10.0.0.52:3306) completed successfully.
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步