十五、MySQL主从、备份与高可用。
1.编写脚本，支持让用户自主选择，使用mysqldump还是xtraback全量备份。
##实现备份，需要开启数据库二进制日志
[root@localhost ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
log-bin
[root@localhost ~]# cat mysqlbackup.sh
#!/bin/bash
#
. /etc/init.d/functions
username=root
password=123456
PS3="Please input a number[1|2]: "
mysqldumpdir=/mysqldumpdir
xtrabackupdir=/xtrabackupdir-`date +"%F-%T"`
[ -d $mysqldumpdir ] || mkdir $mysqldumpdir
select backup_method in mysqldump xtrabackup;do
case $backup_method in
mysqldump)
mysqldump -u${username} -p${password} -A -F -E -R -q \
--single-transaction --master-data=1 --flush-privileges \
--triggers --default-character-set=utf8 \
> $mysqldumpdir/`date +"%F-%T"`.sql && \
action "`date +"%F-%T"` data backup to $mysqldumpdir" true || action "backup data" false
break
;;
xtrabackup)
xtrabackup --user=$username --password=$password \
--backup --target-dir=$xtrabackupdir \
&> /dev/null && action "`date +"%F-%T"` data backup to $xtrabackupdir" true \
|| action "backup data" false
break
;;
*)
echo "not support"
;;
esac
done
[root@localhost ~]# bash +x mysqlbackup.sh
1) mysqldump
2) xtrabackup
Please input a number[1|2]: 2
backup data [FAILED]
[root@localhost ~]# bash +x mysqlbackup.sh
1) mysqldump
2) xtrabackup
Please input a number[1|2]: 1
2021-04-14-14:36:33 data backup to /mysqldumpdir [ OK ]
2.配置Mysql主从同步
master服务器：10.50.100.7
slave服务器：10.50.100.8
systemctl stop firewalld
setenforce 0
master服务器配置
[root@master ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server-id=7
log-bin
[root@master ~]# systemctl restart mariadb
[root@master ~]# mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.3.27-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> grant replication slave on *.* to repluser@'10.50.100.%' identified by 'magedu';
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> show master logs;
+--------------------+-----------+
| Log_name | File_size |
+--------------------+-----------+
| mariadb-bin.000001 | 379 |
| mariadb-bin.000002 | 424 |
| mariadb-bin.000003 | 424 |
| mariadb-bin.000004 | 398 |
| mariadb-bin.000005 | 530 |
+--------------------+-----------+
5 rows in set (0.000 sec)
Slave服务器配置
[root@slave ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server-id=8
[root@slave ~]# systemctl start mariadb
[root@slave ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.3.27-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> help change master to
MariaDB [(none)]> CHANGE MASTER TO
-> MASTER_HOST='10.50.100.7',
-> MASTER_USER='repluser',
-> MASTER_PASSWORD='magedu',
-> MASTER_PORT=3306,
-> MASTER_LOG_FILE='mariadb-bin.000005',
-> MASTER_LOG_POS=530;
Query OK, 0 rows affected (0.027 sec)
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.50.100.7
Master_User: repluser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mariadb-bin.000005
Read_Master_Log_Pos: 530
Relay_Log_File: mariadb-relay-bin.000002
Relay_Log_Pos: 557
Relay_Master_Log_File: mariadb-bin.000005
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 530
Relay_Log_Space: 868
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 7
Master_SSL_Crl:
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos:
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: conservative
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Slave_DDL_Groups: 0
Slave_Non_Transactional_Groups: 0
Slave_Transactional_Groups: 0
1 row in set (0.000 sec)
3.使用MHA实现Mysql高可用。
服务器环境
1 10.50.100.22 CentOS7 MHA管理端
2 10.50.100.7 CentOS8 Master
3 10.50.100.8 CentOS8 Slave1
4 10.50.100.9 CentOS8 Slave2
实现master
[root@master ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server-id=7
log-bin
skip_name_resolve=1
[root@master ~]# systemctl start mariadb
[root@master ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.3.27-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show master logs;
+--------------------+-----------+
| Log_name | File_size |
+--------------------+-----------+
| mariadb-bin.000001 | 28243 |
| mariadb-bin.000002 | 344 |
+--------------------+-----------+
2 rows in set (0.000 sec)
MariaDB [(none)]> grant replication slave on *.* to repluser@'10.50.100.%' identified by 'magedu';
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> grant all on *.* to mhauser@'10.50.100.%' identified by 'magedu';
Query OK, 0 rows affected (0.001 sec)
实现slave1
[root@slave1 ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server-id=8
log-bin
read_only
relay_log_purge=0
skip_name_resolve=1
[root@slave1 ~]# systemctl start mariadb
[root@slave1 ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.3.27-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> help change master to
MariaDB [(none)]> CHANGE MASTER TO MASTER_HOST='10.50.100.7', MASTER_USER='repluser', MASTER_PASSWORD='magedu', MASTER_PORT=3306, MASTER_LOG_FILE='mariadb-bin.000002', MASTER_LOG_POS=344;
Query OK, 0 rows affected (0.027 sec)
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.50.100.7
Master_User: repluser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mariadb-bin.000002
Read_Master_Log_Pos: 729
Relay_Log_File: mariadb-relay-bin.000002
Relay_Log_Pos: 942
Relay_Master_Log_File: mariadb-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 729
Relay_Log_Space: 1253
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 7
Master_SSL_Crl:
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos:
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: conservative
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Slave_DDL_Groups: 2
Slave_Non_Transactional_Groups: 0
Slave_Transactional_Groups: 0
1 row in set (0.000 sec)
实现slave2
[root@slave2 ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server-id=9
log-bin
read_only
relay_log_purge=0
skip_name_resolve=1
[root@slave2 ~]# systemctl start mariadb
[root@slave2 ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.3.27-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> CHANGE MASTER TO
-> MASTER_HOST='10.50.100.7',
-> MASTER_USER='repluser',
-> MASTER_PASSWORD='magedu',
-> MASTER_PORT=3306,
-> MASTER_LOG_FILE='mariadb-bin.000002',
-> MASTER_LOG_POS=344;
Query OK, 0 rows affected (0.006 sec)
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.50.100.7
Master_User: repluser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mariadb-bin.000002
Read_Master_Log_Pos: 729
Relay_Log_File: mariadb-relay-bin.000002
Relay_Log_Pos: 942
Relay_Master_Log_File: mariadb-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 729
Relay_Log_Space: 1253
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 7
Master_SSL_Crl:
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos:
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: conservative
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Slave_DDL_Groups: 2
Slave_Non_Transactional_Groups: 0
Slave_Transactional_Groups: 0
1 row in set (0.000 sec)
主库检查从库状态
MariaDB [(none)]> show slave hosts;
+-----------+------+------+-----------+
| Server_id | Host | Port | Master_id |
+-----------+------+------+-----------+
| 9 | | 3306 | 7 |
| 8 | | 3306 | 7 |
+-----------+------+------+-----------+
2 rows in set (0.000 sec)
在所有节点实现相互之间ssh key验证
[root@mha-manager ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:pFcT3+S5iuJnvxhXB1O0QogwmpZhNFzdRFmBh+0NTV8 root@mha-manager
The key's randomart image is:
+---[RSA 2048]----+
| o=.+o *oB+=oE|
| ..* .o O.B ++|
| = . o +.O..|
| . o . . ..= |
| . S o .|
| . . o . |
| . o o |
| . .o+ |
| .o..o. |
+----[SHA256]-----+
[root@mha-manager ~]# ll /root/.ssh/
total 8
-rw-------. 1 root root 1675 Apr 15 13:57 id_rsa
-rw-r--r--. 1 root root 398 Apr 15 13:57 id_rsa.pub
[root@mha-manager ~]# ssh-copy-id 10.50.100.22
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.50.100.22 (10.50.100.22)' can't be established.
ECDSA key fingerprint is SHA256:kjA0XhlixSyniVVf5Hx8wA0i0TE5phqPBLPqq1bI4Hs.
ECDSA key fingerprint is MD5:66:ff:32:a7:96:bd:12:0c:cd:f2:07:d2:89:96:f2:ac.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.50.100.22's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '10.50.100.22'"
and check to make sure that only the key(s) you wanted were added.
[root@mha-manager ~]# ll /root/.ssh/
total 16
-rw-------. 1 root root 398 Apr 15 14:01 authorized_keys
-rw-------. 1 root root 1675 Apr 15 13:57 id_rsa
-rw-r--r--. 1 root root 398 Apr 15 13:57 id_rsa.pub
-rw-r--r--. 1 root root 174 Apr 15 13:58 known_hosts
[root@mha-manager ~]# rsync -av .ssh 10.50.100.7:/root/
root@10.50.100.7's password:
sending incremental file list
.ssh/
.ssh/authorized_keys
.ssh/id_rsa
.ssh/id_rsa.pub
.ssh/known_hosts
sent 3,156 bytes received 96 bytes 929.14 bytes/sec
total size is 2,818 speedup is 0.87
[root@mha-manager ~]# rsync -av .ssh 10.50.100.8:/root/
The authenticity of host '10.50.100.8 (10.50.100.8)' can't be established.
ECDSA key fingerprint is SHA256:k9AY1O0h/0rZ/CgtlSCzA4Ckon9UO8scYDhceU6Yxu8.
ECDSA key fingerprint is MD5:cd:f7:28:97:e2:cb:cd:4d:6a:5e:65:44:de:a4:e1:96.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.50.100.8' (ECDSA) to the list of known hosts.
root@10.50.100.8's password:
sending incremental file list
.ssh/
.ssh/authorized_keys
.ssh/id_rsa
.ssh/id_rsa.pub
.ssh/known_hosts
sent 3,330 bytes received 96 bytes 622.91 bytes/sec
total size is 2,991 speedup is 0.87
[root@mha-manager ~]# rsync -av .ssh 10.50.100.9:/root/
The authenticity of host '10.50.100.9 (10.50.100.9)' can't be established.
ECDSA key fingerprint is SHA256:k9AY1O0h/0rZ/CgtlSCzA4Ckon9UO8scYDhceU6Yxu8.
ECDSA key fingerprint is MD5:cd:f7:28:97:e2:cb:cd:4d:6a:5e:65:44:de:a4:e1:96.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.50.100.9' (ECDSA) to the list of known hosts.
root@10.50.100.9's password:
sending incremental file list
.ssh/
.ssh/authorized_keys
.ssh/id_rsa
.ssh/id_rsa.pub
.ssh/known_hosts
sent 3,503 bytes received 96 bytes 654.36 bytes/sec
total size is 3,164 speedup is 0.88
验证免密登录
[root@mha-manager ~]# ssh root@10.50.100.7
Last login: Wed Apr 14 17:28:28 2021 from 10.0.0.110
[root@master ~]# exit
logout
Connection to 10.50.100.7 closed.
[root@mha-manager ~]# ssh root@10.50.100.8
Last login: Wed Apr 14 17:28:44 2021 from 10.0.0.110
[root@slave1 ~]# exit
logout
Connection to 10.50.100.8 closed.
[root@mha-manager ~]# ssh root@10.50.100.9
Last login: Wed Apr 14 17:29:02 2021 from 10.0.0.110
[root@slave2 ~]# exit
logout
Connection to 10.50.100.9 closed.
安装yum扩展包，所有服务器都安装
[root@mha-manager ~]# wget http://mirrors.sohu.com/fedora-epel/epel-release-latest-7.noarch.rpm
[root@mha-manager ~]# yum install -y epel-release-latest-7.noarch.rpm
[root@master ~]# wget http://mirrors.sohu.com/fedora-epel/epel-release-latest-7.noarch.rpm
[root@master ~]# yum install -y epel-release-latest-7.noarch.rpm
[root@slave1 ~]# wget http://mirrors.sohu.com/fedora-epel/epel-release-latest-7.noarch.rpm
[root@slave1 ~]# yum install -y epel-release-latest-7.noarch.rpm
[root@slave2 ~]# wget http://mirrors.sohu.com/fedora-epel/epel-release-latest-7.noarch.rpm
[root@slave2 ~]# yum install -y epel-release-latest-7.noarch.rpm
修改 /etc/yum.repos.d/epel.repo 文件参数
把所有服务器 /etc/yum.repos.d/epel.repo 文件中的 gpgcheck 参数值设置成 0 。
[root@mha-manager ~]# vim /etc/yum.repos.d/epel.repo
gpgcheck=0
所有服务器安装所需要的依赖包
[root@mha-manager ~]# yum list perl-DBD-MySQL ncftp perl-DBI
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* epel: mirrors.bfsu.edu.cn
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Available Packages
ncftp.x86_64 2:3.2.5-7.el7 epel
perl-DBD-MySQL.x86_64 4.023-6.el7 base
perl-DBI.x86_64 1.627-4.el7 base
[root@mha-manager ~]# yum install -y perl-DBD-MySQL ncftp perl-DBI
安装监控服务器依赖包
[root@mha-manager ~]# yum install -y perl-Config-Tiny.noarch perl-Time-HiRes.x86_64 perl-Parallel-ForkManager perl-Log-Dispatch.noarch
在管理节点上安装两个包(不支持CentOS8，只支持CentOS7 以下版本)
[root@mha-manager ~]# yum -y install mha*.rpm
Loaded plugins: fastestmirror
Examining mha4mysql-manager-0.57-0.el7.noarch.rpm: mha4mysql-manager-0.57-0.el7.noarch
Marking mha4mysql-manager-0.57-0.el7.noarch.rpm to be installed
Examining mha4mysql-node-0.57-0.el7.noarch.rpm: mha4mysql-node-0.57-0.el7.noarch
Marking mha4mysql-node-0.57-0.el7.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package mha4mysql-manager.noarch 0:0.57-0.el7 will be installed
---> Package mha4mysql-node.noarch 0:0.57-0.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================================
Installing:
mha4mysql-manager noarch 0.57-0.el7 /mha4mysql-manager-0.57-0.el7.noarch 327 k
mha4mysql-node noarch 0.57-0.el7 /mha4mysql-node-0.57-0.el7.noarch 103 k
Transaction Summary
==============================================================================================================================================================================
Install 2 Packages
Total size: 430 k
Installed size: 430 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : mha4mysql-node-0.57-0.el7.noarch 1/2
Installing : mha4mysql-manager-0.57-0.el7.noarch 2/2
Verifying : mha4mysql-node-0.57-0.el7.noarch 1/2
Verifying : mha4mysql-manager-0.57-0.el7.noarch 2/2
Installed:
mha4mysql-manager.noarch 0:0.57-0.el7 mha4mysql-node.noarch 0:0.57-0.el7
Complete!
在被管理节点安装mha4mysql-node（支持CentOS 8，7，6）
[root@master ~]# yum -y install mha4mysql-node-0.57-0.el7.noarch.rpm
[root@slave1 ~]# yum -y install mha4mysql-node-0.57-0.el7.noarch.rpm
[root@slave2 ~]# yum -y install mha4mysql-node-0.57-0.el7.noarch.rpm
在管理节点建立配置文件
[root@mha-manager ~]# mkdir /etc/mastermha/
[root@mha-manager ~]# vim /etc/mastermha/app1.conf
[server default]
user=mhauser
password=magedu
manager_workdir=/data/mastermha/app1/
manager_log=/data/mastermha/app1/manager.log
remote_workdir=/data/mastermha/app1/
ssh_user=root
repl_user=repluser
repl_password=magedu
ping_interval=1
[server1]
hostname=10.50.100.7
candidate_master=1
[server2]
hostname=10.50.100.8
candidate_master=1
[server3]
hostname=10.50.100.9
检查Mha的环境
[root@mha-manager ~]# masterha_check_ssh --conf=/etc/mastermha/app1.conf
Thu Apr 15 14:41:01 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Apr 15 14:41:01 2021 - [info] Reading application default configuration from /etc/mastermha/app1.conf..
Thu Apr 15 14:41:01 2021 - [info] Reading server configuration from /etc/mastermha/app1.conf..
Thu Apr 15 14:41:01 2021 - [info] Starting SSH connection tests..
Thu Apr 15 14:41:02 2021 - [debug]
Thu Apr 15 14:41:01 2021 - [debug] Connecting via SSH from root@10.50.100.7(10.50.100.7:22) to root@10.50.100.8(10.50.100.8:22)..
Warning: Permanently added '10.50.100.8' (ECDSA) to the list of known hosts.
Thu Apr 15 14:41:01 2021 - [debug] ok.
Thu Apr 15 14:41:01 2021 - [debug] Connecting via SSH from root@10.50.100.7(10.50.100.7:22) to root@10.50.100.9(10.50.100.9:22)..
Warning: Permanently added '10.50.100.9' (ECDSA) to the list of known hosts.
Thu Apr 15 14:41:02 2021 - [debug] ok.
Thu Apr 15 14:41:03 2021 - [debug]
Thu Apr 15 14:41:02 2021 - [debug] Connecting via SSH from root@10.50.100.9(10.50.100.9:22) to root@10.50.100.7(10.50.100.7:22)..
Thu Apr 15 14:41:02 2021 - [debug] ok.
Thu Apr 15 14:41:02 2021 - [debug] Connecting via SSH from root@10.50.100.9(10.50.100.9:22) to root@10.50.100.8(10.50.100.8:22)..
Thu Apr 15 14:41:03 2021 - [debug] ok.
Thu Apr 15 14:41:03 2021 - [debug]
Thu Apr 15 14:41:01 2021 - [debug] Connecting via SSH from root@10.50.100.8(10.50.100.8:22) to root@10.50.100.7(10.50.100.7:22)..
Thu Apr 15 14:41:02 2021 - [debug] ok.
Thu Apr 15 14:41:02 2021 - [debug] Connecting via SSH from root@10.50.100.8(10.50.100.8:22) to root@10.50.100.9(10.50.100.9:22)..
Warning: Permanently added '10.50.100.9' (ECDSA) to the list of known hosts.
Thu Apr 15 14:41:02 2021 - [debug] ok.
Thu Apr 15 14:41:03 2021 - [info] All SSH connection tests passed successfully.
[root@mha-manager ~]# masterha_check_repl --conf=/etc/mastermha/app1.conf
Thu Apr 15 14:41:15 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Apr 15 14:41:15 2021 - [info] Reading application default configuration from /etc/mastermha/app1.conf..
Thu Apr 15 14:41:15 2021 - [info] Reading server configuration from /etc/mastermha/app1.conf..
Thu Apr 15 14:41:15 2021 - [info] MHA::MasterMonitor version 0.57.
Creating directory /data/mastermha/app1/.. done.
Thu Apr 15 14:41:17 2021 - [info] GTID failover mode = 0
Thu Apr 15 14:41:17 2021 - [info] Dead Servers:
Thu Apr 15 14:41:17 2021 - [info] Alive Servers:
Thu Apr 15 14:41:17 2021 - [info] 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:41:17 2021 - [info] 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:41:17 2021 - [info] 10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:41:17 2021 - [info] Alive Slaves:
Thu Apr 15 14:41:17 2021 - [info] 10.50.100.8(10.50.100.8:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:41:17 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:41:17 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Apr 15 14:41:17 2021 - [info] 10.50.100.9(10.50.100.9:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:41:17 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:41:17 2021 - [info] Current Alive Master: 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:41:17 2021 - [info] Checking slave configurations..
Thu Apr 15 14:41:17 2021 - [info] Checking replication filtering settings..
Thu Apr 15 14:41:17 2021 - [info] binlog_do_db= , binlog_ignore_db=
Thu Apr 15 14:41:17 2021 - [info] Replication filtering check ok.
Thu Apr 15 14:41:17 2021 - [info] GTID (with auto-pos) is not supported
Thu Apr 15 14:41:17 2021 - [info] Starting SSH connection tests..
Thu Apr 15 14:41:19 2021 - [info] All SSH connection tests passed successfully.
Thu Apr 15 14:41:19 2021 - [info] Checking MHA Node version..
Thu Apr 15 14:41:20 2021 - [info] Version check ok.
Thu Apr 15 14:41:20 2021 - [info] Checking SSH publickey authentication settings on the current master..
Thu Apr 15 14:41:20 2021 - [info] HealthCheck: SSH to 10.50.100.7 is reachable.
Thu Apr 15 14:41:20 2021 - [info] Master MHA Node version is 0.57.
Thu Apr 15 14:41:20 2021 - [info] Checking recovery script configurations on 10.50.100.7(10.50.100.7:3306)..
Thu Apr 15 14:41:20 2021 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.57 --start_file=mariadb-bin.000002
Thu Apr 15 14:41:20 2021 - [info] Connecting to root@10.50.100.7(10.50.100.7:22)..
Creating /data/mastermha/app1 if not exists.. Creating directory /data/mastermha/app1.. done.
ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to mariadb-bin.000002
Thu Apr 15 14:41:21 2021 - [info] Binlog setting check done.
Thu Apr 15 14:41:21 2021 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Thu Apr 15 14:41:21 2021 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=10.50.100.8 --slave_ip=10.50.100.8 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=10.3.27-MariaDB-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Thu Apr 15 14:41:21 2021 - [info] Connecting to root@10.50.100.8(10.50.100.8:22)..
Creating directory /data/mastermha/app1/.. done.
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Apr 15 14:41:21 2021 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=10.50.100.9 --slave_ip=10.50.100.9 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=10.3.27-MariaDB-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Thu Apr 15 14:41:21 2021 - [info] Connecting to root@10.50.100.9(10.50.100.9:22)..
Creating directory /data/mastermha/app1/.. done.
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Apr 15 14:41:22 2021 - [info] Slaves settings check done.
Thu Apr 15 14:41:22 2021 - [info]
10.50.100.7(10.50.100.7:3306) (current master)
+--10.50.100.8(10.50.100.8:3306)
+--10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:41:22 2021 - [info] Checking replication health on 10.50.100.8..
Thu Apr 15 14:41:22 2021 - [info] ok.
Thu Apr 15 14:41:22 2021 - [info] Checking replication health on 10.50.100.9..
Thu Apr 15 14:41:22 2021 - [info] ok.
Thu Apr 15 14:41:22 2021 - [warning] master_ip_failover_script is not defined.
Thu Apr 15 14:41:22 2021 - [warning] shutdown_script is not defined.
Thu Apr 15 14:41:22 2021 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
启动MHA
##默认前台运行
[root@mha-manager ~]# nohup masterha_manager --conf=/etc/mastermha/app1.conf &> /dev/null
##查看状态
[root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.conf
app1 (pid:49733) is running(0:PING_OK), master:10.50.100.7
排错日志
[root@mha-manager ~]# tail /data/mastermha/app1/manager.log
10.50.100.7(10.50.100.7:3306) (current master)
+--10.50.100.8(10.50.100.8:3306)
+--10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:44:19 2021 - [warning] master_ip_failover_script is not defined.
Thu Apr 15 14:44:19 2021 - [warning] shutdown_script is not defined.
Thu Apr 15 14:44:19 2021 - [info] Set master ping interval 1 seconds.
Thu Apr 15 14:44:19 2021 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Thu Apr 15 14:44:19 2021 - [info] Starting ping health check on 10.50.100.7(10.50.100.7:3306)..
Thu Apr 15 14:44:19 2021 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
模拟故障：master DOWN
[root@mha-manager ~]# masterha_manager --conf=/etc/mastermha/app1.conf
Thu Apr 15 14:49:22 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Apr 15 14:49:22 2021 - [info] Reading application default configuration from /etc/mastermha/app1.conf..
Thu Apr 15 14:49:22 2021 - [info] Reading server configuration from /etc/mastermha/app1.conf..
[root@mha-manager ~]# cat /data/mastermha/app1/manager.log
Thu Apr 15 14:44:14 2021 - [info] MHA::MasterMonitor version 0.57.
Thu Apr 15 14:44:15 2021 - [info] GTID failover mode = 0
Thu Apr 15 14:44:15 2021 - [info] Dead Servers:
Thu Apr 15 14:44:15 2021 - [info] Alive Servers:
Thu Apr 15 14:44:15 2021 - [info] 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:44:15 2021 - [info] 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:44:15 2021 - [info] 10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:44:15 2021 - [info] Alive Slaves:
Thu Apr 15 14:44:15 2021 - [info] 10.50.100.8(10.50.100.8:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:44:15 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:44:15 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Apr 15 14:44:15 2021 - [info] 10.50.100.9(10.50.100.9:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:44:15 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:44:15 2021 - [info] Current Alive Master: 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:44:15 2021 - [info] Checking slave configurations..
Thu Apr 15 14:44:15 2021 - [info] Checking replication filtering settings..
Thu Apr 15 14:44:15 2021 - [info] binlog_do_db= , binlog_ignore_db=
Thu Apr 15 14:44:15 2021 - [info] Replication filtering check ok.
Thu Apr 15 14:44:15 2021 - [info] GTID (with auto-pos) is not supported
Thu Apr 15 14:44:15 2021 - [info] Starting SSH connection tests..
Thu Apr 15 14:44:17 2021 - [info] All SSH connection tests passed successfully.
Thu Apr 15 14:44:17 2021 - [info] Checking MHA Node version..
Thu Apr 15 14:44:18 2021 - [info] Version check ok.
Thu Apr 15 14:44:18 2021 - [info] Checking SSH publickey authentication settings on the current master..
Thu Apr 15 14:44:18 2021 - [info] HealthCheck: SSH to 10.50.100.7 is reachable.
Thu Apr 15 14:44:18 2021 - [info] Master MHA Node version is 0.57.
Thu Apr 15 14:44:18 2021 - [info] Checking recovery script configurations on 10.50.100.7(10.50.100.7:3306)..
Thu Apr 15 14:44:18 2021 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.57 --start_file=mariadb-bin.000002
Thu Apr 15 14:44:18 2021 - [info] Connecting to root@10.50.100.7(10.50.100.7:22)..
Creating /data/mastermha/app1 if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to mariadb-bin.000002
Thu Apr 15 14:44:19 2021 - [info] Binlog setting check done.
Thu Apr 15 14:44:19 2021 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Thu Apr 15 14:44:19 2021 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=10.50.100.8 --slave_ip=10.50.100.8 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=10.3.27-MariaDB-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Thu Apr 15 14:44:19 2021 - [info] Connecting to root@10.50.100.8(10.50.100.8:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Apr 15 14:44:19 2021 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=10.50.100.9 --slave_ip=10.50.100.9 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=10.3.27-MariaDB-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Thu Apr 15 14:44:19 2021 - [info] Connecting to root@10.50.100.9(10.50.100.9:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Apr 15 14:44:19 2021 - [info] Slaves settings check done.
Thu Apr 15 14:44:19 2021 - [info]
10.50.100.7(10.50.100.7:3306) (current master)
+--10.50.100.8(10.50.100.8:3306)
+--10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:44:19 2021 - [warning] master_ip_failover_script is not defined.
Thu Apr 15 14:44:19 2021 - [warning] shutdown_script is not defined.
Thu Apr 15 14:44:19 2021 - [info] Set master ping interval 1 seconds.
Thu Apr 15 14:44:19 2021 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Thu Apr 15 14:44:19 2021 - [info] Starting ping health check on 10.50.100.7(10.50.100.7:3306)..
Thu Apr 15 14:44:19 2021 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Thu Apr 15 14:48:48 2021 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Thu Apr 15 14:48:48 2021 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.57 --binlog_prefix=mariadb-bin
Thu Apr 15 14:48:48 2021 - [info] HealthCheck: SSH to 10.50.100.7 is reachable.
Thu Apr 15 14:48:49 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.50.100.7' (111))
Thu Apr 15 14:48:49 2021 - [warning] Connection failed 2 time(s)..
Thu Apr 15 14:48:50 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.50.100.7' (111))
Thu Apr 15 14:48:50 2021 - [warning] Connection failed 3 time(s)..
Thu Apr 15 14:48:51 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.50.100.7' (111))
Thu Apr 15 14:48:51 2021 - [warning] Connection failed 4 time(s)..
Thu Apr 15 14:48:51 2021 - [warning] Master is not reachable from health checker!
Thu Apr 15 14:48:51 2021 - [warning] Master 10.50.100.7(10.50.100.7:3306) is not reachable!
Thu Apr 15 14:48:51 2021 - [warning] SSH is reachable.
Thu Apr 15 14:48:51 2021 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mastermha/app1.conf again, and trying to connect to all servers to check server status..
Thu Apr 15 14:48:51 2021 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Apr 15 14:48:51 2021 - [info] Reading application default configuration from /etc/mastermha/app1.conf..
Thu Apr 15 14:48:51 2021 - [info] Reading server configuration from /etc/mastermha/app1.conf..
Thu Apr 15 14:48:52 2021 - [info] GTID failover mode = 0
Thu Apr 15 14:48:52 2021 - [info] Dead Servers:
Thu Apr 15 14:48:52 2021 - [info] 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:52 2021 - [info] Alive Servers:
Thu Apr 15 14:48:52 2021 - [info] 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:48:52 2021 - [info] 10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:48:52 2021 - [info] Alive Slaves:
Thu Apr 15 14:48:52 2021 - [info] 10.50.100.8(10.50.100.8:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:52 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:52 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Apr 15 14:48:52 2021 - [info] 10.50.100.9(10.50.100.9:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:52 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:52 2021 - [info] Checking slave configurations..
Thu Apr 15 14:48:52 2021 - [info] Checking replication filtering settings..
Thu Apr 15 14:48:52 2021 - [info] Replication filtering check ok.
Thu Apr 15 14:48:52 2021 - [info] Master is down!
Thu Apr 15 14:48:52 2021 - [info] Terminating monitoring script.
Thu Apr 15 14:48:52 2021 - [info] Got exit code 20 (Master dead).
Thu Apr 15 14:48:52 2021 - [info] MHA::MasterFailover version 0.57.
Thu Apr 15 14:48:52 2021 - [info] Starting master failover.
Thu Apr 15 14:48:52 2021 - [info]
Thu Apr 15 14:48:52 2021 - [info] * Phase 1: Configuration Check Phase..
Thu Apr 15 14:48:52 2021 - [info]
Thu Apr 15 14:48:53 2021 - [info] GTID failover mode = 0
Thu Apr 15 14:48:53 2021 - [info] Dead Servers:
Thu Apr 15 14:48:53 2021 - [info] 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:53 2021 - [info] Checking master reachability via MySQL(double check)...
Thu Apr 15 14:48:53 2021 - [info] ok.
Thu Apr 15 14:48:53 2021 - [info] Alive Servers:
Thu Apr 15 14:48:53 2021 - [info] 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:48:53 2021 - [info] 10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:48:53 2021 - [info] Alive Slaves:
Thu Apr 15 14:48:53 2021 - [info] 10.50.100.8(10.50.100.8:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:53 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:53 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Apr 15 14:48:53 2021 - [info] 10.50.100.9(10.50.100.9:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:53 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:53 2021 - [info] Starting Non-GTID based failover.
Thu Apr 15 14:48:53 2021 - [info]
Thu Apr 15 14:48:53 2021 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Apr 15 14:48:53 2021 - [info]
Thu Apr 15 14:48:53 2021 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Apr 15 14:48:53 2021 - [info]
Thu Apr 15 14:48:53 2021 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Apr 15 14:48:53 2021 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
Thu Apr 15 14:48:53 2021 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Apr 15 14:48:54 2021 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Thu Apr 15 14:48:54 2021 - [info]
Thu Apr 15 14:48:54 2021 - [info] * Phase 3: Master Recovery Phase..
Thu Apr 15 14:48:54 2021 - [info]
Thu Apr 15 14:48:54 2021 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Apr 15 14:48:54 2021 - [info]
Thu Apr 15 14:48:54 2021 - [info] The latest binary log file/position on all slaves is mariadb-bin.000002:729
Thu Apr 15 14:48:54 2021 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Apr 15 14:48:54 2021 - [info] 10.50.100.8(10.50.100.8:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:54 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:54 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Apr 15 14:48:54 2021 - [info] 10.50.100.9(10.50.100.9:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:54 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:54 2021 - [info] The oldest binary log file/position on all slaves is mariadb-bin.000002:729
Thu Apr 15 14:48:54 2021 - [info] Oldest slaves:
Thu Apr 15 14:48:54 2021 - [info] 10.50.100.8(10.50.100.8:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:54 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:54 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Apr 15 14:48:54 2021 - [info] 10.50.100.9(10.50.100.9:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:54 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:54 2021 - [info]
Thu Apr 15 14:48:54 2021 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Thu Apr 15 14:48:54 2021 - [info]
Thu Apr 15 14:48:54 2021 - [info] Fetching dead master's binary logs..
Thu Apr 15 14:48:54 2021 - [info] Executing command on the dead master 10.50.100.7(10.50.100.7:3306): save_binary_logs --command=save --start_file=mariadb-bin.000002 --start_pos=729 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57
Creating /data/mastermha/app1 if not exists.. ok.
Concat binary/relay logs from mariadb-bin.000002 pos 729 to mariadb-bin.000002 EOF into /data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog ..
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 256.. ok.
Dumping effective binlog data from /var/lib/mysql/mariadb-bin.000002 position 729 to tail(752).. ok.
Binlog Checksum enabled
Concat succeeded.
Thu Apr 15 14:48:55 2021 - [info] scp from root@10.50.100.7:/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog to local:/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog succeeded.
Thu Apr 15 14:48:55 2021 - [info] HealthCheck: SSH to 10.50.100.8 is reachable.
Thu Apr 15 14:48:56 2021 - [info] HealthCheck: SSH to 10.50.100.9 is reachable.
Thu Apr 15 14:48:56 2021 - [info]
Thu Apr 15 14:48:56 2021 - [info] * Phase 3.3: Determining New Master Phase..
Thu Apr 15 14:48:56 2021 - [info]
Thu Apr 15 14:48:56 2021 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Thu Apr 15 14:48:56 2021 - [info] All slaves received relay logs to the same position. No need to resync each other.
Thu Apr 15 14:48:56 2021 - [info] Searching new master from slaves..
Thu Apr 15 14:48:56 2021 - [info] Candidate masters from the configuration file:
Thu Apr 15 14:48:56 2021 - [info] 10.50.100.8(10.50.100.8:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:48:56 2021 - [info] Replicating from 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:48:56 2021 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Apr 15 14:48:56 2021 - [info] Non-candidate masters:
Thu Apr 15 14:48:56 2021 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Thu Apr 15 14:48:56 2021 - [info] New master is 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:48:56 2021 - [info] Starting master failover..
Thu Apr 15 14:48:56 2021 - [info]
From:
10.50.100.7(10.50.100.7:3306) (current master)
+--10.50.100.8(10.50.100.8:3306)
+--10.50.100.9(10.50.100.9:3306)
To:
10.50.100.8(10.50.100.8:3306) (new master)
+--10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:48:56 2021 - [info]
Thu Apr 15 14:48:56 2021 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Thu Apr 15 14:48:56 2021 - [info]
Thu Apr 15 14:48:56 2021 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Apr 15 14:48:56 2021 - [info] Sending binlog..
Thu Apr 15 14:48:57 2021 - [info] scp from local:/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog to root@10.50.100.8:/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog succeeded.
Thu Apr 15 14:48:57 2021 - [info]
Thu Apr 15 14:48:57 2021 - [info] * Phase 3.4: Master Log Apply Phase..
Thu Apr 15 14:48:57 2021 - [info]
Thu Apr 15 14:48:57 2021 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Thu Apr 15 14:48:57 2021 - [info] Starting recovery on 10.50.100.8(10.50.100.8:3306)..
Thu Apr 15 14:48:57 2021 - [info] Generating diffs succeeded.
Thu Apr 15 14:48:57 2021 - [info] Waiting until all relay logs are applied.
Thu Apr 15 14:48:57 2021 - [info] done.
Thu Apr 15 14:48:57 2021 - [info] Getting slave status..
Thu Apr 15 14:48:57 2021 - [info] This slave(10.50.100.8)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mariadb-bin.000002:729). No need to recover from Exec_Master_Log_Pos.
Thu Apr 15 14:48:57 2021 - [info] Connecting to the target slave host 10.50.100.8, running recover script..
Thu Apr 15 14:48:57 2021 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mhauser' --slave_host=10.50.100.8 --slave_ip=10.50.100.8 --slave_port=3306 --apply_files=/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog --workdir=/data/mastermha/app1/ --target_version=10.3.27-MariaDB-log --timestamp=20210415144852 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --slave_pass=xxx
Thu Apr 15 14:48:57 2021 - [info]
MySQL client version is 10.3.27. Using --binary-mode.
Applying differential binary/relay log files /data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog on 10.50.100.8:3306. This may take long time...
Applying log files succeeded.
Thu Apr 15 14:48:57 2021 - [info] All relay logs were successfully applied.
Thu Apr 15 14:48:57 2021 - [info] Getting new master's binlog name and position..
Thu Apr 15 14:48:57 2021 - [info] mariadb-bin.000003:344
Thu Apr 15 14:48:57 2021 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.50.100.8', MASTER_PORT=3306, MASTER_LOG_FILE='mariadb-bin.000003', MASTER_LOG_POS=344, MASTER_USER='repluser', MASTER_PASSWORD='xxx';
Thu Apr 15 14:48:57 2021 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Thu Apr 15 14:48:57 2021 - [info] Setting read_only=0 on 10.50.100.8(10.50.100.8:3306)..
Thu Apr 15 14:48:57 2021 - [info] ok.
Thu Apr 15 14:48:57 2021 - [info] ** Finished master recovery successfully.
Thu Apr 15 14:48:57 2021 - [info] * Phase 3: Master Recovery Phase completed.
Thu Apr 15 14:48:57 2021 - [info]
Thu Apr 15 14:48:57 2021 - [info] * Phase 4: Slaves Recovery Phase..
Thu Apr 15 14:48:57 2021 - [info]
Thu Apr 15 14:48:57 2021 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Thu Apr 15 14:48:57 2021 - [info]
Thu Apr 15 14:48:57 2021 - [info] -- Slave diff file generation on host 10.50.100.9(10.50.100.9:3306) started, pid: 50105. Check tmp log /data/mastermha/app1//10.50.100.9_3306_20210415144852.log if it takes time..
Thu Apr 15 14:48:58 2021 - [info]
Thu Apr 15 14:48:58 2021 - [info] Log messages from 10.50.100.9 ...
Thu Apr 15 14:48:58 2021 - [info]
Thu Apr 15 14:48:57 2021 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Apr 15 14:48:58 2021 - [info] End of log messages from 10.50.100.9.
Thu Apr 15 14:48:58 2021 - [info] -- 10.50.100.9(10.50.100.9:3306) has the latest relay log events.
Thu Apr 15 14:48:58 2021 - [info] Generating relay diff files from the latest slave succeeded.
Thu Apr 15 14:48:58 2021 - [info]
Thu Apr 15 14:48:58 2021 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Thu Apr 15 14:48:58 2021 - [info]
Thu Apr 15 14:48:58 2021 - [info] -- Slave recovery on host 10.50.100.9(10.50.100.9:3306) started, pid: 50107. Check tmp log /data/mastermha/app1//10.50.100.9_3306_20210415144852.log if it takes time..
Thu Apr 15 14:48:59 2021 - [info]
Thu Apr 15 14:48:59 2021 - [info] Log messages from 10.50.100.9 ...
Thu Apr 15 14:48:59 2021 - [info]
Thu Apr 15 14:48:58 2021 - [info] Sending binlog..
Thu Apr 15 14:48:59 2021 - [info] scp from local:/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog to root@10.50.100.9:/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog succeeded.
Thu Apr 15 14:48:59 2021 - [info] Starting recovery on 10.50.100.9(10.50.100.9:3306)..
Thu Apr 15 14:48:59 2021 - [info] Generating diffs succeeded.
Thu Apr 15 14:48:59 2021 - [info] Waiting until all relay logs are applied.
Thu Apr 15 14:48:59 2021 - [info] done.
Thu Apr 15 14:48:59 2021 - [info] Getting slave status..
Thu Apr 15 14:48:59 2021 - [info] This slave(10.50.100.9)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mariadb-bin.000002:729). No need to recover from Exec_Master_Log_Pos.
Thu Apr 15 14:48:59 2021 - [info] Connecting to the target slave host 10.50.100.9, running recover script..
Thu Apr 15 14:48:59 2021 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mhauser' --slave_host=10.50.100.9 --slave_ip=10.50.100.9 --slave_port=3306 --apply_files=/data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog --workdir=/data/mastermha/app1/ --target_version=10.3.27-MariaDB-log --timestamp=20210415144852 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --slave_pass=xxx
Thu Apr 15 14:48:59 2021 - [info]
MySQL client version is 10.3.27. Using --binary-mode.
Applying differential binary/relay log files /data/mastermha/app1//saved_master_binlog_from_10.50.100.7_3306_20210415144852.binlog on 10.50.100.9:3306. This may take long time...
Applying log files succeeded.
Thu Apr 15 14:48:59 2021 - [info] All relay logs were successfully applied.
Thu Apr 15 14:48:59 2021 - [info] Resetting slave 10.50.100.9(10.50.100.9:3306) and starting replication from the new master 10.50.100.8(10.50.100.8:3306)..
Thu Apr 15 14:48:59 2021 - [info] Executed CHANGE MASTER.
Thu Apr 15 14:48:59 2021 - [info] Slave started.
Thu Apr 15 14:48:59 2021 - [info] End of log messages from 10.50.100.9.
Thu Apr 15 14:48:59 2021 - [info] -- Slave recovery on host 10.50.100.9(10.50.100.9:3306) succeeded.
Thu Apr 15 14:48:59 2021 - [info] All new slave servers recovered successfully.
Thu Apr 15 14:48:59 2021 - [info]
Thu Apr 15 14:48:59 2021 - [info] * Phase 5: New master cleanup phase..
Thu Apr 15 14:48:59 2021 - [info]
Thu Apr 15 14:48:59 2021 - [info] Resetting slave info on the new master..
Thu Apr 15 14:48:59 2021 - [info] 10.50.100.8: Resetting slave info succeeded.
Thu Apr 15 14:48:59 2021 - [info] Master failover to 10.50.100.8(10.50.100.8:3306) completed successfully.
Thu Apr 15 14:48:59 2021 - [info]
----- Failover Report -----
app1: MySQL Master failover 10.50.100.7(10.50.100.7:3306) to 10.50.100.8(10.50.100.8:3306) succeeded
Master 10.50.100.7(10.50.100.7:3306) is down!
Check MHA Manager logs at mha-manager:/data/mastermha/app1/manager.log for details.
Started automated(non-interactive) failover.
The latest slave 10.50.100.8(10.50.100.8:3306) has all relay logs for recovery.
Selected 10.50.100.8(10.50.100.8:3306) as a new master.
10.50.100.8(10.50.100.8:3306): OK: Applying all logs succeeded.
10.50.100.9(10.50.100.9:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
10.50.100.9(10.50.100.9:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.50.100.8(10.50.100.8:3306)
10.50.100.8(10.50.100.8:3306): Resetting slave info succeeded.
Master failover to 10.50.100.8(10.50.100.8:3306) completed successfully.
Thu Apr 15 14:49:22 2021 - [info] MHA::MasterMonitor version 0.57.
Thu Apr 15 14:49:23 2021 - [info] GTID failover mode = 0
Thu Apr 15 14:49:23 2021 - [info] Dead Servers:
Thu Apr 15 14:49:23 2021 - [info] 10.50.100.7(10.50.100.7:3306)
Thu Apr 15 14:49:23 2021 - [info] Alive Servers:
Thu Apr 15 14:49:23 2021 - [info] 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:49:23 2021 - [info] 10.50.100.9(10.50.100.9:3306)
Thu Apr 15 14:49:23 2021 - [info] Alive Slaves:
Thu Apr 15 14:49:23 2021 - [info] 10.50.100.9(10.50.100.9:3306) Version=10.3.27-MariaDB-log (oldest major version between slaves) log-bin:enabled
Thu Apr 15 14:49:23 2021 - [info] Replicating from 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:49:23 2021 - [info] Current Alive Master: 10.50.100.8(10.50.100.8:3306)
Thu Apr 15 14:49:23 2021 - [info] Checking slave configurations..
Thu Apr 15 14:49:23 2021 - [info] Checking replication filtering settings..
Thu Apr 15 14:49:23 2021 - [info] binlog_do_db= , binlog_ignore_db=
Thu Apr 15 14:49:23 2021 - [info] Replication filtering check ok.
Thu Apr 15 14:49:23 2021 - [info] GTID (with auto-pos) is not supported
Thu Apr 15 14:49:23 2021 - [info] Starting SSH connection tests..
Thu Apr 15 14:49:24 2021 - [info] All SSH connection tests passed successfully.
Thu Apr 15 14:49:24 2021 - [info] Checking MHA Node version..
Thu Apr 15 14:49:25 2021 - [info] Version check ok.
Thu Apr 15 14:49:25 2021 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln492] Server 10.50.100.7(10.50.100.7:3306) is dead, but must be alive! Check server settings.
Thu Apr 15 14:49:25 2021 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line 402.
Thu Apr 15 14:49:25 2021 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.
Thu Apr 15 14:49:25 2021 - [info] Got exit code 1 (Not master dead).
[root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.conf
app1 is stopped(2:NOT_RUNNING).
