mha-mysql切换

二.手动Failover（MHA Manager必须没有运行）

手动failover，这种场景意味着在业务上没有启用MHA自动切换功能，当主服务器故障时，人工手动调用MHA来进行故障切换操作，具体命令如下：

注意：如果，MHA manager检测到没有dead的server，将报错，并结束failover： 

Mon Apr 21 21:23:33 2014 - [info] Dead Servers:
Mon Apr 21 21:23:33 2014 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln181] None of server is dead. Stop failover.
Mon Apr 21 21:23:33 2014 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln178] Got ERROR:  at /usr/local/bin/masterha_master_switch line 53

进行手动切换命令如下：

输出的信息会询问你是否进行切换：

Mon Apr 21 21:28:00 2014 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Apr 21 21:28:00 2014 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Mon Apr 21 21:28:00 2014 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Mon Apr 21 21:28:00 2014 - [info] MHA::MasterFailover version 0.53.
Mon Apr 21 21:28:00 2014 - [info] Starting master failover.
Mon Apr 21 21:28:00 2014 - [info] 
Mon Apr 21 21:28:00 2014 - [info] * Phase 1: Configuration Check Phase..
Mon Apr 21 21:28:00 2014 - [info] 
Mon Apr 21 21:28:00 2014 - [info] Dead Servers:
Mon Apr 21 21:28:00 2014 - [info]   192.168.0.50(192.168.0.50:3306)
Mon Apr 21 21:28:00 2014 - [info] Checking master reachability via mysql(double check)..
Mon Apr 21 21:28:00 2014 - [info]  ok.
Mon Apr 21 21:28:00 2014 - [info] Alive Servers:
Mon Apr 21 21:28:00 2014 - [info]   192.168.0.60(192.168.0.60:3306)
Mon Apr 21 21:28:00 2014 - [info]   192.168.0.70(192.168.0.70:3306)
Mon Apr 21 21:28:00 2014 - [info] Alive Slaves:
Mon Apr 21 21:28:00 2014 - [info]   192.168.0.60(192.168.0.60:3306)  Version=5.5.19-ndb-7.2.4-gpl-log (oldest major version between slaves) log-bin:enabled
Mon Apr 21 21:28:00 2014 - [info]     Replicating from 192.168.0.50(192.168.0.50:3306)
Mon Apr 21 21:28:00 2014 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Apr 21 21:28:00 2014 - [info]   192.168.0.70(192.168.0.70:3306)  Version=5.5.19-ndb-7.2.4-gpl-log (oldest major version between slaves) log-bin:enabled
Mon Apr 21 21:28:00 2014 - [info]     Replicating from 192.168.0.50(192.168.0.50:3306)
Master 192.168.0.50 is dead. Proceed? (yes/NO): yes
Mon Apr 21 21:36:01 2014 - [info] ** Phase 1: Configuration Check Phase completed.
Mon Apr 21 21:36:01 2014 - [info] 
Mon Apr 21 21:36:01 2014 - [info] * Phase 2: Dead Master Shutdown Phase..
Mon Apr 21 21:36:01 2014 - [info] 
Mon Apr 21 21:36:01 2014 - [info] HealthCheck: SSH to 192.168.0.50 is reachable.
Mon Apr 21 21:36:01 2014 - [info] Forcing shutdown so that applications never connect to the current master..
Mon Apr 21 21:36:01 2014 - [info] Executing master IP deactivatation script:
Mon Apr 21 21:36:01 2014 - [info]   /usr/local/bin/master_ip_failover --orig_master_host=192.168.0.50 --orig_master_ip=192.168.0.50 --orig_master_port=3306 --command=stopssh --ssh_user=root  


IN SCRIPT TEST====/sbin/ifconfig eth1:1 down==/sbin/ifconfig eth1:1 192.168.0.88/24===

Disabling the VIP on old master: 192.168.0.50 
Mon Apr 21 21:36:02 2014 - [info]  done.
Mon Apr 21 21:36:02 2014 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Mon Apr 21 21:36:02 2014 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Mon Apr 21 21:36:02 2014 - [info] 
Mon Apr 21 21:36:02 2014 - [info] * Phase 3: Master Recovery Phase..
Mon Apr 21 21:36:02 2014 - [info] 
Mon Apr 21 21:36:02 2014 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Mon Apr 21 21:36:02 2014 - [info] 
Mon Apr 21 21:36:02 2014 - [info] The latest binary log file/position on all slaves is mysql-bin.000020:112
Mon Apr 21 21:36:02 2014 - [info] Latest slaves (Slaves that received relay log files to the latest):
Mon Apr 21 21:36:02 2014 - [info]   192.168.0.60(192.168.0.60:3306)  Version=5.5.19-ndb-7.2.4-gpl-log (oldest major version between slaves) log-bin:enabled
Mon Apr 21 21:36:02 2014 - [info]     Replicating from 192.168.0.50(192.168.0.50:3306)
Mon Apr 21 21:36:02 2014 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Apr 21 21:36:02 2014 - [info]   192.168.0.70(192.168.0.70:3306)  Version=5.5.19-ndb-7.2.4-gpl-log (oldest major version between slaves) log-bin:enabled
Mon Apr 21 21:36:02 2014 - [info]     Replicating from 192.168.0.50(192.168.0.50:3306)
Mon Apr 21 21:36:02 2014 - [info] The oldest binary log file/position on all slaves is mysql-bin.000020:112
Mon Apr 21 21:36:02 2014 - [info] Oldest slaves:
Mon Apr 21 21:36:02 2014 - [info]   192.168.0.60(192.168.0.60:3306)  Version=5.5.19-ndb-7.2.4-gpl-log (oldest major version between slaves) log-bin:enabled
Mon Apr 21 21:36:02 2014 - [info]     Replicating from 192.168.0.50(192.168.0.50:3306)
Mon Apr 21 21:36:02 2014 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Apr 21 21:36:02 2014 - [info]   192.168.0.70(192.168.0.70:3306)  Version=5.5.19-ndb-7.2.4-gpl-log (oldest major version between slaves) log-bin:enabled
Mon Apr 21 21:36:02 2014 - [info]     Replicating from 192.168.0.50(192.168.0.50:3306)
Mon Apr 21 21:36:02 2014 - [info] 
Mon Apr 21 21:36:02 2014 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Mon Apr 21 21:36:02 2014 - [info] 
Mon Apr 21 21:36:02 2014 - [info] Fetching dead master's binary logs..
Mon Apr 21 21:36:02 2014 - [info] Executing command on the dead master 192.168.0.50(192.168.0.50:3306): save_binary_logs --command=save --start_file=mysql-bin.000020  --start_pos=112 --binlog_dir=/data/mysql --output_file=/tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.53
  Creating /tmp if not exists..    ok.
 Concat binary/relay logs from mysql-bin.000020 pos 112 to mysql-bin.000020 EOF into /tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog ..
  Dumping binlog format description event, from position 0 to 112.. ok.
  Dumping effective binlog data from /data/mysql/mysql-bin.000020 position 112 to tail(131).. ok.
 Concat succeeded.
saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog                                                   100%  131     0.1KB/s   00:00    
Mon Apr 21 21:36:02 2014 - [info] scp from root@192.168.0.50:/tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog to local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog succeeded.
Mon Apr 21 21:36:02 2014 - [info] HealthCheck: SSH to 192.168.0.60 is reachable.
Mon Apr 21 21:36:03 2014 - [info] HealthCheck: SSH to 192.168.0.70 is reachable.
Mon Apr 21 21:36:03 2014 - [info] 
Mon Apr 21 21:36:03 2014 - [info] * Phase 3.3: Determining New Master Phase..
Mon Apr 21 21:36:03 2014 - [info] 
Mon Apr 21 21:36:03 2014 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Mon Apr 21 21:36:03 2014 - [info] All slaves received relay logs to the same position. No need to resync each other.
Mon Apr 21 21:36:03 2014 - [info] 192.168.0.60 can be new master.
Mon Apr 21 21:36:03 2014 - [info] New master is 192.168.0.60(192.168.0.60:3306)
Mon Apr 21 21:36:03 2014 - [info] Starting master failover..
Mon Apr 21 21:36:03 2014 - [info] 
From:
192.168.0.50 (current master)
 +--192.168.0.60
 +--192.168.0.70

To:
192.168.0.60 (new master)
 +--192.168.0.70

Starting master switch from 192.168.0.50(192.168.0.50:3306) to 192.168.0.60(192.168.0.60:3306)? (yes/NO): yes
Mon Apr 21 21:36:06 2014 - [info] New master decided manually is 192.168.0.60(192.168.0.60:3306)
Mon Apr 21 21:36:06 2014 - [info] 
Mon Apr 21 21:36:06 2014 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Mon Apr 21 21:36:06 2014 - [info] 
Mon Apr 21 21:36:06 2014 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Mon Apr 21 21:36:06 2014 - [info] Sending binlog..
saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog                                                   100%  131     0.1KB/s   00:00    
Mon Apr 21 21:36:07 2014 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog to root@192.168.0.60:/tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog succeeded.
Mon Apr 21 21:36:07 2014 - [info] 
Mon Apr 21 21:36:07 2014 - [info] * Phase 3.4: Master Log Apply Phase..
Mon Apr 21 21:36:07 2014 - [info] 
Mon Apr 21 21:36:07 2014 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Mon Apr 21 21:36:07 2014 - [info] Starting recovery on 192.168.0.60(192.168.0.60:3306)..
Mon Apr 21 21:36:07 2014 - [info]  Generating diffs succeeded.
Mon Apr 21 21:36:07 2014 - [info] Waiting until all relay logs are applied.
Mon Apr 21 21:36:07 2014 - [info]  done.
Mon Apr 21 21:36:07 2014 - [info] Getting slave status..
Mon Apr 21 21:36:07 2014 - [info] This slave(192.168.0.60)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000020:112). No need to recover from Exec_Master_Log_Pos.
Mon Apr 21 21:36:07 2014 - [info] Connecting to the target slave host 192.168.0.60, running recover script..
Mon Apr 21 21:36:07 2014 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=root --slave_host=192.168.0.60 --slave_ip=192.168.0.60  --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog --workdir=/tmp --target_version=5.5.19-ndb-7.2.4-gpl-log --timestamp=20140421212800 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.53 --slave_pass=xxx
Mon Apr 21 21:36:07 2014 - [info] 
Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog on 192.168.0.60:3306. This may take long time...
Applying log files succeeded.
Mon Apr 21 21:36:07 2014 - [info]  All relay logs were successfully applied.
Mon Apr 21 21:36:07 2014 - [info] Getting new master's binlog name and position..
Mon Apr 21 21:36:07 2014 - [info]  mysql-bin.000022:506716
Mon Apr 21 21:36:07 2014 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.0.60', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000022', MASTER_LOG_POS=506716, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Mon Apr 21 21:36:07 2014 - [info] Executing master IP activate script:
Mon Apr 21 21:36:07 2014 - [info]   /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.0.50 --orig_master_ip=192.168.0.50 --orig_master_port=3306 --new_master_host=192.168.0.60 --new_master_ip=192.168.0.60 --new_master_port=3306  


IN SCRIPT TEST====/sbin/ifconfig eth1:1 down==/sbin/ifconfig eth1:1 192.168.0.88/24===

Enabling the VIP - 192.168.0.88/24 on the new master - 192.168.0.60 
Mon Apr 21 21:36:08 2014 - [info]  OK.
Mon Apr 21 21:36:08 2014 - [info] Setting read_only=0 on 192.168.0.60(192.168.0.60:3306)..
Mon Apr 21 21:36:08 2014 - [info]  ok.
Mon Apr 21 21:36:08 2014 - [info] ** Finished master recovery successfully.
Mon Apr 21 21:36:08 2014 - [info] * Phase 3: Master Recovery Phase completed.
Mon Apr 21 21:36:08 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info] * Phase 4: Slaves Recovery Phase..
Mon Apr 21 21:36:08 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Mon Apr 21 21:36:08 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info] -- Slave diff file generation on host 192.168.0.70(192.168.0.70:3306) started, pid: 33518. Check tmp log /var/log/masterha/app1.log/192.168.0.70_3306_20140421212800.log if it takes time..
Mon Apr 21 21:36:08 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info] Log messages from 192.168.0.70 ...
Mon Apr 21 21:36:08 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Mon Apr 21 21:36:08 2014 - [info] End of log messages from 192.168.0.70.
Mon Apr 21 21:36:08 2014 - [info] -- 192.168.0.70(192.168.0.70:3306) has the latest relay log events.
Mon Apr 21 21:36:08 2014 - [info] Generating relay diff files from the latest slave succeeded.
Mon Apr 21 21:36:08 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Mon Apr 21 21:36:08 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info] -- Slave recovery on host 192.168.0.70(192.168.0.70:3306) started, pid: 33520. Check tmp log /var/log/masterha/app1.log/192.168.0.70_3306_20140421212800.log if it takes time..
saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog                                                   100%  131     0.1KB/s   00:00    
Mon Apr 21 21:36:09 2014 - [info] 
Mon Apr 21 21:36:09 2014 - [info] Log messages from 192.168.0.70 ...
Mon Apr 21 21:36:09 2014 - [info] 
Mon Apr 21 21:36:08 2014 - [info] Sending binlog..
Mon Apr 21 21:36:08 2014 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog to root@192.168.0.70:/tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog succeeded.
Mon Apr 21 21:36:08 2014 - [info] Starting recovery on 192.168.0.70(192.168.0.70:3306)..
Mon Apr 21 21:36:08 2014 - [info]  Generating diffs succeeded.
Mon Apr 21 21:36:08 2014 - [info] Waiting until all relay logs are applied.
Mon Apr 21 21:36:08 2014 - [info]  done.
Mon Apr 21 21:36:08 2014 - [info] Getting slave status..
Mon Apr 21 21:36:08 2014 - [info] This slave(192.168.0.70)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000020:112). No need to recover from Exec_Master_Log_Pos.
Mon Apr 21 21:36:08 2014 - [info] Connecting to the target slave host 192.168.0.70, running recover script..
Mon Apr 21 21:36:08 2014 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=root --slave_host=192.168.0.70 --slave_ip=192.168.0.70  --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog --workdir=/tmp --target_version=5.5.19-ndb-7.2.4-gpl-log --timestamp=20140421212800 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.53 --slave_pass=xxx
Mon Apr 21 21:36:09 2014 - [info] 
Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.0.50_3306_20140421212800.binlog on 192.168.0.70:3306. This may take long time...
Applying log files succeeded.
Mon Apr 21 21:36:09 2014 - [info]  All relay logs were successfully applied.
Mon Apr 21 21:36:09 2014 - [info]  Resetting slave 192.168.0.70(192.168.0.70:3306) and starting replication from the new master 192.168.0.60(192.168.0.60:3306)..
Mon Apr 21 21:36:09 2014 - [info]  Executed CHANGE MASTER.
Mon Apr 21 21:36:09 2014 - [info]  Slave started.
Mon Apr 21 21:36:09 2014 - [info] End of log messages from 192.168.0.70.
Mon Apr 21 21:36:09 2014 - [info] -- Slave recovery on host 192.168.0.70(192.168.0.70:3306) succeeded.
Mon Apr 21 21:36:09 2014 - [info] All new slave servers recovered successfully.
Mon Apr 21 21:36:09 2014 - [info] 
Mon Apr 21 21:36:09 2014 - [info] * Phase 5: New master cleanup phease..
Mon Apr 21 21:36:09 2014 - [info] 
Mon Apr 21 21:36:09 2014 - [info] Resetting slave info on the new master..
Mon Apr 21 21:36:09 2014 - [info]  192.168.0.60: Resetting slave info succeeded.
Mon Apr 21 21:36:09 2014 - [info] Master failover to 192.168.0.60(192.168.0.60:3306) completed successfully.
Mon Apr 21 21:36:09 2014 - [info] 

----- Failover Report -----

app1: MySQL Master failover 192.168.0.50 to 192.168.0.60 succeeded

Master 192.168.0.50 is down!

Check MHA Manager logs at server01 for details.

Started manual(interactive) failover.
Invalidated master IP address on 192.168.0.50.
The latest slave 192.168.0.60(192.168.0.60:3306) has all relay logs for recovery.
Selected 192.168.0.60 as a new master.
192.168.0.60: OK: Applying all logs succeeded.
192.168.0.60: OK: Activated master IP address.
192.168.0.70: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.0.70: OK: Applying all logs succeeded. Slave started, replicating from 192.168.0.60.
192.168.0.60: Resetting slave info succeeded.
Master failover to 192.168.0.60(192.168.0.60:3306) completed successfully.

 

[root@192.168.0.20 ~]# masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf --dead_master_host=192.168.0.50 --dead_master_port=3306 --new_master_host=192.168.0.60 --new_master_port=3306 --ignore_last_failover