|
|
|
|
|
基于 MHA 的 MySQL 高可用方案
一、MHA 简介
MHA(Master High Availability)目前在 MySQL 高可用方面是一个相对成熟的解决方案, 它由日本 DeNA 公司的 youshimaton 员工(现就职于 Facebook 公司)开发,是一套优秀的作为 MySQL 高可用性环境下故障切换和主从角色提升的高可用软件。在 MySQL 故障切换过程中,MHA 能做到在 0~30 秒之内自动完成数据库的主从故障切换操作,并且在进行故障切换的过程中,MHA 能在最大程度上保证数据的一致性,以达到真正意义上的高可用。
MHA 由两部分组成:MHA Manage(r 管理节点)和 MHA Node(数据节点)。MHA Manager
可以单独部署在一台独立的机器上管理多个 master-slave 集群,也可以部署在一台 slave 节点上。MHA Node 运行在每台MySQL 服务器及Manager 服务器上,MHA Manager 会定时探测集群中的 master 节点,当 master 出现故障时,它可以自动将拥有最新数据的 slave 提升为新的 master,然后将所有其他的 slave 重新指向新提升的 master。整个故障转移过程对应用程序层面完全透明。
在 MHA 自动故障切换过程中,MHA 会试图从宕机的主服务器上保存二进制日志,最大程度的保证数据不丢失,但这种操作是有概率性的。例如,如果主服务器硬件故障或无法通过 ssh 访问,MHA 没法保存二进制日志,只进行故障转移从而丢失了最新的数据。使用 MySQL
5.5 的半同步复制,可以降低数据丢失的风险。MHA 可以与半同步复制结合起来。如果只有一个 slave 已经收到了最新的二进制日志,MHA 可以将最新的二进制日志应用于其他所有的slave 服务器上,因此可以保证所有节点的数据一致性。
目前 MHA 主要支持一主多从的架构,要搭建 MHA,要求一个复制集群中必须最少有三台数据库服务器,一主二从,即一台充当 master,一台充当备用 master,另外一台充当从库,因为至少需要三台服务器,出于机器成本的考虑,淘宝也在该基础上进行了改造,目前淘宝 TMHA 已经支持一主一从。另外对于想快速搭建的可以参考:MHA 快速搭建
我们自己使用其实也可以使用 1 主 1 从,但是 master 主机宕机后无法切换,以及无法补全
binlog。master 的 mysqld 进程crash 后,还是可以切换成功,以及补全 binlog 的。官方介绍:https://code.google.com/p/mysql-master-ha/
![](https://img2018.cnblogs.com/blog/1738924/201910/1738924-20191023090237377-1891217566.png)
1.1 、工作流程
(1) 从宕机崩溃的 master 上尝试保存二进制日志事件(binlog events);
(2) 识别含有最新更新的 slave 服务器;
(3) 应用差异的中继日志(relay log)到其他的 slave;
(4) 应用从 master 保存的二进制日志事件(binlog events);
(5) 提升一个 slave 为新的 master 服务器;
(6) 将其他的 slave 连接指向新的 master 进行主从复制;
1.2 、MHA 工具介绍
MHA 软件由两部分组成,Manager 工具包和 Node 工具包,具体的说明如下:
masterha_check_ssh 检查 MHA 的 SSH 配置状况Manager 工具包主要包括以下几个工具:
- masterha_check_repl 检查 MySQL 复制状况
- masterha_manger 启动 MHA
- masterha_check_status 检测当前MHA 运行状态
- masterha_master_monitor 检测 master 是否宕机
- masterha_master_switch 控制故障转移(自动或者手动)
- masterha_conf_host 添加或删除配置的 server 信息
Node 工具包(这些工具通常由 MHA Manager 的脚本触发,无需人为操作)主要包括以下几个工具:
➢ save_binary_logs 保存和复制 master 的二进制日志
➢ apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的
slave
➢ filter_mysqlbinlog 去除不必要的 ROLLBACK 事件(MHA 已不再使用这个工具)
➢ purge_relay_logs 清除中继日志(不会阻塞 SQL 线程)
注意:为了尽可能的减少主库硬件损坏宕机造成的数据丢失,因此在配置 MHA 的同时建议配置成 MySQL 5.5 的半同步复制。关于半同步复制原理各位自己进行查阅。(不是必须)
1.1 、MHA 环境说明
角色 IP 地址 主机名 ServerID 数据库类型
Primary Master 192.168.200.111 server01 1 写入
Secondary Master 192.168.200.112 server02 2 写入
Slave1 192.168.200.113 server03 3 读
Slave2 192.168.200.114 server04 4 读
Manager 192.168.200.115 server05 - 监控复制组
LVS Matser 192.168.200.116 localhost -
LVS Backup 192.168.200.117 localhost -
二、前期环境部署
2.1、配置所有主机名称
master1 主机:
hostname server01
bash
master2 主机:
hostname server02
bash
slave1 主机:
hostname server03
bash
slave2 主机:
hostname server04
bash
manager 主机:
hostname server05
bash
2.2、配置所有主机名映射
[root@server05 ~]# vim /etc/hosts
192.168.200.111 server01
192.168.200.112 server02
192.168.200.113 server03
192.168.200.114 server04
192.168.200.115 server05
scp /etc/hosts 192.168.200.111:/etc/
scp /etc/hosts 192.168.200.112:/etc/
scp /etc/hosts 192.168.200.113:/etc/
scp /etc/hosts 192.168.200.114:/etc/
2.3、所有主机关闭防火墙和安全机制
systemctl stop iptables
systemctl stop firewalld
setenforce 0
2.4、下载 mha-manager 和 mha-node
http://downloads.mariadb.com/MHA/
三、安装 MHA node
3.1、所有主机安装 MHA node 及相关 perl 依赖包
安装 epel 源:
rpm -ivh epel-release-latest-7.noarch.rpm
yum install -y perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder
perl-ExtUtils-MakeMaker
rpm -q perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder
perl-ExtUtils-MakeMaker
perl-DBD-MySQL-4.023-6.el7.x86_64
perl-DBI-1.627-4.el7.x86_64
perl-CPAN-1.9800-292.el7.noarch
perl-ExtUtils-CBuilder-0.28.2.6-292.el7.noarch
perl-ExtUtils-MakeMaker-6.68-3.el7.noarch
注意:安装后建议检查一下所需软件包是否全部安装
3.2、所有主机上安装 MHA Node
tar xf mha4mysql-node-0.56.tar.gz
cd mha4mysql-node-0.56/
perl Makefile.PL
make && make install
3.3、MHA Node 安装完后会在 /usr/local/bin 生成以下脚本
ls -l /usr/local/bin/
总用量 40
-r-xr-xr-x 1 root root 16346 12 月 29 15:07 apply_diff_relay_logs
-r-xr-xr-x 1 root root 4807 12 月 29 15:07 filter_mysqlbinlog
-r-xr-xr-x 1 root root 7401 12 月 29 15:07 purge_relay_logs
-r-xr-xr-x 1 root root 7395 12 月 29 15:07 save_binary_logs
四、安装 MHA Manger
注意:安装 MHA Manger 之前也需要安装 MHA Node
4.1、首先安装 MHA Manger 依赖的 perl 模块(我这里使用 yum 安装)
yum install -y perl perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI
perl-Time-HiRes
yum -y install perl-Config-Tiny-2.14-7.el7.noarch.rpm
rpm -q perl cpan perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI
perl-Time-HiRes perl-Config-Tiny
perl-5.16.3-292.el7.x86_64
perl-Log-Dispatch-2.41-1.el7.1.noarch
perl-Parallel-ForkManager-1.18-2.el7.noarch
perl-DBD-MySQL-4.023-6.el7.x86_64
perl-DBI-1.627-4.el7.x86_64
perl-Time-HiRes-1.9725-3.el7.x86_64
perl-Config-Tiny-2.14-7.el7.noarch
注意:之前时候 perl-Config-Tiny.noarch 没有安装成功,后来用 cpan(cpan install Config::Tiny )
4.2、安装 MHA Manger 软件包
tar xf mha4mysql-manager-0.56.tar.gz
cd mha4mysql-manager-0.56/
perl Makefile.PL
make && make install
4.3、安装完成后会有以下脚本文件
ls -l /usr/local/bin/
总用量 76
-r-xr-xr-x 1 root root 16346 12 月 29 15:08 apply_diff_relay_logs
-r-xr-xr-x 1 root root 4807 12 月 29 15:08 filter_mysqlbinlog
-r-xr-xr-x 1 root root 1995 12 月 29 15:37 masterha_check_repl
-r-xr-xr-x 1 root root 1779 12 月 29 15:37 masterha_check_ssh
-r-xr-xr-x 1 root root 1865 12 月 29 15:37 masterha_check_status
-r-xr-xr-x 1 root root 3201 12 月 29 15:37 masterha_conf_host
-r-xr-xr-x 1 root root 2517 12 月 29 15:37 masterha_manager
-r-xr-xr-x 1 root root 2165 12 月 29 15:37 masterha_master_monitor
-r-xr-xr-x 1 root root 2373 12 月 29 15:37 masterha_master_switch
-r-xr-xr-x 1 root root 3879 12 月 29 15:37 masterha_secondary_check
-r-xr-xr-x 1 root root 1739 12 月 29 15:37 masterha_stop
-r-xr-xr-x 1 root root 7401 12 月 29 15:08 purge_relay_logs
-r-xr-xr-x 1 root root 7395 12 月 29 15:08 save_binary_logs
五、配置 SSH 密钥对验证
服务器之间需要实现密钥对验证。关于配置密钥对验证可看下面步骤。但是有一点需要
注意:不能禁止 password 登陆,否则会出现错误
1. 服务器先生成一个密钥对
2. 把自己的公钥传给对方
5.1、Server05(192.168.200.115)上:
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114
注意:Server05 需要连接每个主机测试,因为第一次连接的时候需要输入 yes,影响后期故
障切换时,对于每个主机的 SSH 控制。
ssh server01
ssh server02
ssh server03
ssh server04
5.2、Primary Master(192.168.200.111):
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114
5.3、Secondary Master(192.168.200.112):
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114
5.4、slave1(192.168.200.113):
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114
5.5、slave2(192.168.200.114):
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113
六、安装 mysql
111-114 主机上的操作:
yum -y install mariadb mariadb-server mariadb-devel
systemctl start mariadb
netstat -lnpt | grep :3306
设置数据库初始密码(后续操作中使用)
mysqladmin -u root password 123456
七、搭建主从复制环境
注意:binlog-do-db 和 replicate-ignore-db 设置必须相同。 MHA 在启动时候会检测过滤规
则,如果过滤规则不同,MHA 将不启动监控和故障转移功能。
7.1、修改 mysql 主机的配置文件
Primary Master(192.168.200.111):
vim /etc/my.cnf
[mysqld]
server-id = 1
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb
Secondary Master(192.168.200.112):
vim /etc/my.cnf
[mysqld]
server-id=2
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb
slave1(192.168.200.113):
vim /etc/my.cnf
[mysqld]
server-id=3
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb
slave2(192.168.200.114):
vim /etc/my.cnf
[mysqld]
server-id=4
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb
7.2、在 Primary Master(192.168.200.111)上对旧数据进行备份 新机器不用作
mysqldump --master-data=2 --single-transaction -R --triggers -A > all.sql
参数解释:
--master-data=2 备份时刻记录 master 的 Binlog 位置和 Position
--single-transaction 获取一致性快照
-R 备份存储过程和函数
-triggres 备份触发器
-A 备份所有的库。
7.3、mysql 服务器创建复制授权用户
grant replication slave on *.* to 'repl'@'192.168.200.%' identified by '123456';
flush privileges;
7.4、查看主库备份时的 binlog 名称和位置
MariaDB [(none)]> show master status;
+-------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-bin.000001 | 474 | | |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)
7.5、把数据备份复制其他主机
scp all.sql 192.168.200.112:/tmp/
scp all.sql 192.168.200.113:/tmp/
scp all.sql 192.168.200.114:/tmp/
7.6、导入备份数据到 112-114 主机上执行复制相关命令
mysql -uroot -p123456< /tmp/all.sql
stop slave;
CHANGE MASTER TO
MASTER_HOST='192.168.200.111',
MASTER_USER='repl',
MASTER_PASSWORD='123456',
MASTER_LOG_FILE='master-bin.000001',
MASTER_LOG_POS=474;
start slave;
show slave status\G
# 检查 IO 和 SQL 线程是否为:yes
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
7.7、主从同步故障处理
Slave_IO_Running: No
Slave_SQL_Running: Yes
-----------------------------------忽略部分信息-----------------------------------
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from
binary log: 'Could not find first log file name in binary log index file'
-----------------------------------忽略部分信息-----------------------------------
处理方式:
stop slave;
reset slave;
set global sql_slave_skip_counter =1 ;
start slave;
7.8、三台 slave 服务器设置 read_only 状态
从库对外只提供读服务,只所以没有写进 mysql 配置文件,是因为随时 server02 会提升为
master
[root@server02 ~]# mysql -uroot -p123456 -e 'set global read_only=1'
[root@server03 ~]# mysql -uroot -p123456 -e 'set global read_only=1'
[root@server04 ~]# mysql -uroot -p123456 -e 'set global read_only=1'
7.9、创建监控用户(111-114 主机上的操作):
grant all privileges on *.* to 'root'@'192.168.200.%' identified by '123456';
flush privileges;
为自己的主机名授权:
grant all privileges on *.* to 'root'@'server04' identified by '123456';
flush privileges;
到这里整个 mysql 主从集群环境已经搭建完毕。
八、配置 MHA 环境
8.1、创建 MHA 的工作目录及相关配置文件
Server05(192.168.200.115):在软件包解压后的目录里面有样例配置文件
mkdir /etc/masterha
cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha
8.2、修改 app1.cnf 配置文件
/usr/local/bin/master_ip_failover 脚本需要根据自己环境修改 ip 和网卡名称等。
vim /etc/masterha/app1.cnf
[server default]
#设置 manager 的工作日志
manager_workdir=/var/log/masterha/app1
#设置 manager 的日志,这两条都是默认存在的
manager_log=/var/log/masterha/app1/manager.log
#设置 master 默认保存 binlog 的位置,以便 MHA 可以找到 master 日志
master_binlog_dir=/var/lib/mysql
#设置自动 failover 时候的切换脚本
master_ip_failover_script= /usr/local/bin/master_ip_failover
#设置 mysql 中 root 用户的密码
password=123456
user=root
#ping 包的时间间隔
ping_interval=1
#设置远端 mysql 在发生切换时保存 binlog 的具体位置
remote_workdir=/tmp
#设置复制用户的密码和用户名
repl_password=123456
repl_user=repl
[server1]
hostname=server01
port=3306
[server2]
hostname=server02
candidate_master=1
port=3306
check_repl_delay=0
[server3]
hostname=server03
port=3306
[server4]
hostname=server04
port=3306
8.3、配置故障转移脚本
[root@server05 ~]# vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,
);
my $vip = '192.168.200.100'; # 写入 VIP
my $key = "1"; #非 keepalived 方式切换脚本使用的
my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令
$ssh_user = "root";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
#eval {
# print "Disabling the VIP on old master: $orig_master_host \n";
# &stop_vip();
# $exit_code = 0;
#};
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
#my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F',' '{print $3}' | awk '{print $1}'`;
#if ( $ping le "90.0%"&& $ping gt "0.0%" ){
#$exit_code = 0;
#}
#else {
&stop_vip();
# updating global catalog, etc
$exit_code = 0;
#}
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $ssh_user\@$orig_master_ip \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host
--orig_master_ip=ip --orig_master_port=port --
new_master_host=host --new_master_ip=ip --new_master_port=port\n"; }
[root@server05 ~]# chmod +x /usr/local/bin/master_ip_failover
8.4、设置从库 relay log 的清除方式(112-114):
mysql -uroot -p123456 -e 'set global relay_log_purge=0;'
注意:
MHA 在故障切换的过程中,从库的恢复过程依赖于 relay log 的相关信息,所以这里要
将 relay log 的自动清除设置为 OFF,采用手动清除 relay log 的方式。在默认情况下,从服务
器上的中继日志会在 SQL 线程执行完毕后被自动删除。但是在 MHA 环境中,这些中继日志
在恢复其他从服务器时可能会被用到,因此需要禁用中继日志的自动清除功能。定期清除中
继日志需要考虑到复制延时的问题。在 ext3 的文件系统下,删除大的文件需要一定的时间,
会导致严重的复制延时。为了避免复制延时,需要暂时为中继日志创建硬链接,因为在 linux
系统中通过硬链接删除大文件速度会很快。(在 mysql 数据库中,删除大表时,通常也采用
建立硬链接的方式)
8.5、配置从库(112-114)relay_log 清除脚本加入计划任务
MHA 节点中包含了 pure_relay_logs 命令工具,它可以为中继日志创建硬链接,执行 SET
GLOBAL relay_log_purge=1,等待几秒钟以便 SQL 线程切换到新的中继日志,再执行 SET
GLOBAL relay_log_purge=0。
vim purge_relay_log.sh
#!/bin/bash
user=root
passwd=123456 #注意:数据库要有密码,填自己所设置的密码就可以,前面设置过
port=3306
log_dir='/tmp'
work_dir='/tmp'
purge='/usr/local/bin/purge_relay_logs'
if [ ! -d $log_dir ]
then
mkdir $log_dir -p
fi
$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port
--workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1
chmod +x purge_relay_log.sh
crontab -e
0 4 * * * /bin/bash /root/purge_relay_log.sh
pure_relay_logs 脚本参数如下所示:
--user mysql 用户名
--password mysql 密码
--port 端口号
--workdir 指定创建 relay log 的硬链接的位置,默认是/var/tmp,由于系统不同
分区创建硬链接文件会失败,故需要执行硬链接具体位置,成功执行脚本后,硬链接的中继
日志文件被删除
--disable_relay_log_purge 默认情况下,如果 relay_log_purge=1,脚本会什么都不清理,自
动退出,通过设定这个参数,当 relay_log_purge=1 的情况下会将 relay_log_purge 设置为 0。
清理 relay log 之后,最后将参数设置为 OFF。
8.6、手动清除中继日志
purge_relay_logs --user=root --password=123456 --disable_relay_log_purge --port=3306
--workdir=/tmp
2017-08-31 21:33:52: purge_relay_logs script started.
Found relay_log.info: /usr/local/mysql/data/relay-log.info
Removing hard linked relay log files slave-relay-bin* under /tmp.. done.
Current relay log file: /usr/local/mysql/data/slave-relay-bin.000002
Archiving unused relay log files (up to /usr/local/mysql/data/slave-relay-bin.000001) ...
Creating hard link for /usr/local/mysql/data/slave-relay-bin.000001 under
/tmp/slave-relay-bin.000001 .. ok.
Creating hard links for unused relay log files completed.
Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL
thread can delete older relay log files (if i
t keeps up); SET GLOBAL relay_log_purge=0; .. ok. Removing hard linked relay log files
slave-relay-bin* under /tmp.. done.
2017-08-31 21:33:56: All relay log purging operations succeeded.
8.7、检查 MHA ssh 通信状态
[root@server05 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not
found. Skipping.
Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from
/etc/masterha/app1.cnf..
Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests..
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from
root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22)..
Sat Dec 29 16:03:59 2018 - [debug] ok.
Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from
root@server02(192.168.200.112:22) to root@server03(192.168.200.113:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server02(192.168.200.112:22) to root@server04(192.168.200.114:22)..
Sat Dec 29 16:04:02 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from
root@server03(192.168.200.113:22) to root@server01(192.168.200.111:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server03(192.168.200.113:22) to root@server02(192.168.200.112:22)..
Sat Dec 29 16:04:01 2018 - [debug] ok.
Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from
root@server03(192.168.200.113:22) to root@server04(192.168.200.114:22)..
Sat Dec 29 16:04:02 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:57 2018 - [debug] Connecting via SSH from
root@server01(192.168.200.111:22) to root@server02(192.168.200.112:22)..
Sat Dec 29 16:03:59 2018 - [debug] ok.
Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from
root@server01(192.168.200.111:22) to root@server03(192.168.200.113:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server01(192.168.200.111:22) to root@server04(192.168.200.114:22)..
Sat Dec 29 16:04:01 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from
root@server04(192.168.200.114:22) to root@server01(192.168.200.111:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server04(192.168.200.114:22) to root@server02(192.168.200.112:22)..
Sat Dec 29 16:04:01 2018 - [debug] ok.
Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from
root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22)..
Sat Dec 29 16:04:02 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully.
最后会返回 successfully 表示没有问题
8.9、检查整个集群的状态
[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not
found. Skipping.
Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from
/etc/masterha/app1.cnf..
Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests..
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from
root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22)..
Sat Dec 29 16:03:59 2018 - [debug] ok.
Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from
root@server02(192.168.200.112:22) to root@server03(192.168.200.113:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server02(192.168.200.112:22) to root@server04(192.168.200.114:22)..
Sat Dec 29 16:04:02 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from
root@server03(192.168.200.113:22) to root@server01(192.168.200.111:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server03(192.168.200.113:22) to root@server02(192.168.200.112:22)..
Sat Dec 29 16:04:01 2018 - [debug] ok.
Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from
root@server03(192.168.200.113:22) to root@server04(192.168.200.114:22)..
Sat Dec 29 16:04:02 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:57 2018 - [debug] Connecting via SSH from
root@server01(192.168.200.111:22) to root@server02(192.168.200.112:22)..
Sat Dec 29 16:03:59 2018 - [debug] ok.
Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from
root@server01(192.168.200.111:22) to root@server03(192.168.200.113:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server01(192.168.200.111:22) to root@server04(192.168.200.114:22)..
Sat Dec 29 16:04:01 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from
root@server04(192.168.200.114:22) to root@server01(192.168.200.111:22)..
Sat Dec 29 16:04:00 2018 - [debug] ok.
Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from
root@server04(192.168.200.114:22) to root@server02(192.168.200.112:22)..
Sat Dec 29 16:04:01 2018 - [debug] ok.
Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from
root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22)..
Sat Dec 29 16:04:02 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully.
[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Sat Dec 29 16:04:53 2018 - [warning] Global configuration file /etc/masterha_default.cnf not
found. Skipping.
Sat Dec 29 16:04:53 2018 - [info] Reading application default configurations from
/etc/masterha/app1.cnf..
Sat Dec 29 16:04:53 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Sat Dec 29 16:04:53 2018 - [info] MHA::MasterMonitor version 0.56.
Creating directory /var/log/masterha/app1.. done.
Sat Dec 29 16:04:55 2018 - [info] Dead Servers:
Sat Dec 29 16:04:55 2018 - [info] Alive Servers:
Sat Dec 29 16:04:55 2018 - [info] server01(192.168.200.111:3306)
Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306)
Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306)
Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306)
Sat Dec 29 16:04:55 2018 - [info] Alive Slaves:
Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB
(oldest major version between slaves) log-bin:enabled
Sat Dec 29 16:04:55 2018 - [info] Replicating from
192.168.200.111(192.168.200.111:3306)
Sat Dec 29 16:04:55 2018 - [info] Primary candidate for the new Master
(candidate_master is set)
Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB
(oldest major version between slaves) log-bin:enabled
Sat Dec 29 16:04:55 2018 - [info] Replicating from
192.168.200.111(192.168.200.111:3306)
Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB
(oldest major version between slaves) log-bin:enabled
Sat Dec 29 16:04:55 2018 - [info] Replicating from
192.168.200.111(192.168.200.111:3306)
Sat Dec 29 16:04:55 2018 - [info] Current Alive Master: server01(192.168.200.111:3306)
Sat Dec 29 16:04:55 2018 - [info] Checking slave configurations..
Sat Dec 29 16:04:55 2018 - [info] Checking replication filtering settings..
Sat Dec 29 16:04:55 2018 - [info] binlog_do_db= , binlog_ignore_db=
Sat Dec 29 16:04:55 2018 - [info] Replication filtering check ok.
Sat Dec 29 16:04:55 2018 - [info] Starting SSH connection tests..
Sat Dec 29 16:05:00 2018 - [info] All SSH connection tests passed successfully.
Sat Dec 29 16:05:00 2018 - [info] Checking MHA Node version..
Sat Dec 29 16:05:02 2018 - [info] Version check ok.
Sat Dec 29 16:05:02 2018 - [info] Checking SSH publickey authentication settings on the current
master..
Sat Dec 29 16:05:03 2018 - [info] HealthCheck: SSH to server01 is reachable.
Sat Dec 29 16:05:04 2018 - [info] Master MHA Node version is 0.56.
Sat Dec 29 16:05:04 2018 - [info] Checking recovery script configurations on the current
master..
Sat Dec 29 16:05:04 2018 - [info] Executing command: save_binary_logs --command=test
--start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --
manager_version=0.56 --start_file=master-bin.000001 Sat Dec 29 16:05:04 2018 - [info]
Connecting to root@server01(server01)..
Creating /tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to master-bin.000001
Sat Dec 29 16:05:04 2018 - [info] Master setting check done.
Sat Dec 29 16:05:04 2018 - [info] Checking SSH publickey authentication and checking recovery
script configurations on all alive slave servers..
Sat Dec 29 16:05:04 2018 - [info] Executing command : apply_diff_relay_logs
--command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.112
--slave_po
rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56
--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/
--slave_pass=xxxSat Dec 29 16:05:04 2018 - [info] Connecting to
root@192.168.200.112(server02:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Dec 29 16:05:05 2018 - [info] Executing command : apply_diff_relay_logs
--command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.113
--slave_po
rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56
--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/
--slave_pass=xxxSat Dec 29 16:05:05 2018 - [info] Connecting to
root@192.168.200.113(server03:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Dec 29 16:05:07 2018 - [info] Executing command : apply_diff_relay_logs
--command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.114
--slave_po
rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56
--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/
--slave_pass=xxxSat Dec 29 16:05:07 2018 - [info] Connecting to
root@192.168.200.114(server04:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Dec 29 16:05:08 2018 - [info] Slaves settings check done.
Sat Dec 29 16:05:09 2018 - [info]
server01 (current master)
+--server02
+--server03
+--server04
Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server02..
Sat Dec 29 16:05:09 2018 - [info] ok.
Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server03..
Sat Dec 29 16:05:09 2018 - [info] ok.
Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server04..
Sat Dec 29 16:05:09 2018 - [info] ok.
Sat Dec 29 16:05:09 2018 - [info] Checking master_ip_failover_script status:
Sat Dec 29 16:05:09 2018 - [info] /usr/local/bin/master_ip_failover --command=status
--ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m
aster_port=3306
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Checking the Status of the script.. OK
bash: /etc/init.d/keepalived: 没有那个文件或目录
Sat Dec 29 16:05:09 2018 - [info] OK.
Sat Dec 29 16:05:09 2018 - [warning] shutdown_script is not defined.
Sat Dec 29 16:05:09 2018 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
九、VIP 配置管理
Master vip 配置有两种方式,一种是通过 keepalived 或者 heartbeat 类似的软件的方式管
理 VIP 的浮动,另一种为通过命令方式管理。
通过命令方式管理 VIP 地址:
打开在前面编辑过的文件/etc/masterha/app1.cnf,检查如下行是否正确,再检查集群状态。
[root@server05 ~]# grep -n 'master_ip_failover_script' /etc/masterha/app1.cnf
9:master_ip_failover_script= /usr/local/bin/master_ip_failover
Primary Master(192.168.200.111)
[root@server01 ~]# ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group
default qlen 1000
inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32
inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1
Server05(192.168.200.115)修改故障转移脚本
[root@server05 ~]# head -13 /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,
);
my $vip = '192.168.200.100'; # 写入 VIP
my $key = "1"; #非 keepalived 方式切换脚本使用的
my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip"; #若是使用 keepalived
my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令
/usr/local/bin/master_ip_failover 文件的内容意思是当主库发生故障时,会触发 MHA 切
换,MHA manager 会停掉主库上的 ens32:1 接口,触发虚拟 ip 漂移到备选从库,从而完成
切换。
Server05(192.168.200.115) 检查 manager 状态
masterha_check_status --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).
注意:如果正常会显示"PING_OK",否则会显示"NOT_RUNNING",代表 MHA 监控没有开启。
Server05(192.168.200.115) 开启 manager 监控
[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dea
d_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1
&
[1] 65837
启动参数介绍:
--remove_dead_master_conf 该参数代表当发生主从切换后,老的主库的 ip 将会从配置
文件中移除。
--manger_log 日志存放位置
--ignore_last_failover 在缺省情况下,如果 MHA 检测到连续发生宕机,且两次宕
机间隔不足 8 小时的话,则不会进行 Failover,之所以这样限制是为了避免 ping-pong 效应。
该参数代表忽略上次 MHA 触发切换产生的文件,默认情况下,MHA 发生切换后会在日志目
录,也就是上面我设置的/data 产生 app1.failover.complete 文件,下次再次切换的时候如果
发现该目录下存在该文件将不允许触发切换,除非在第一次切换后收到删除该文件,为了方
便,这里设置为--ignore_last_failover。
Server05(192.168.200.115)查看 Server05 监控是否正常:
[root@monitor ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:65837) is running(0:PING_OK), master:server01
可以看见已经在监控了
Server05(192.168.200.115)查看启动日志
[root@server05 ~]# cat /var/log/masterha/app1/manager.log
Sat Dec 29 16:09:50 2018 - [warning] Global configuration file /etc/masterha_default.cnf not
found. Skipping.
Sat Dec 29 16:09:50 2018 - [info] Reading application default configurations from
/etc/masterha/app1.cnf..
Sat Dec 29 16:09:50 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Sat Dec 29 16:09:50 2018 - [info] MHA::MasterMonitor version 0.56.
Sat Dec 29 16:09:51 2018 - [info] Dead Servers:
Sat Dec 29 16:09:51 2018 - [info] Alive Servers:
Sat Dec 29 16:09:51 2018 - [info] server01(192.168.200.111:3306)
Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306)
Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306)
Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306)
Sat Dec 29 16:09:51 2018 - [info] Alive Slaves:
Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB
(oldest major version between slaves) log-bin:enabled
Sat Dec 29 16:09:51 2018 - [info] Replicating from
192.168.200.111(192.168.200.111:3306)
Sat Dec 29 16:09:51 2018 - [info] Primary candidate for the new Master
(candidate_master is set)
Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB
(oldest major version between slaves) log-bin:enabled
Sat Dec 29 16:09:51 2018 - [info] Replicating from
192.168.200.111(192.168.200.111:3306)
Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB
(oldest major version between slaves) log-bin:enabled
Sat Dec 29 16:09:51 2018 - [info] Replicating from
192.168.200.111(192.168.200.111:3306)
Sat Dec 29 16:09:51 2018 - [info] Current Alive Master: server01(192.168.200.111:3306)
Sat Dec 29 16:09:51 2018 - [info] Checking slave configurations..
Sat Dec 29 16:09:51 2018 - [info] Checking replication filtering settings..
Sat Dec 29 16:09:51 2018 - [info] binlog_do_db= , binlog_ignore_db=
Sat Dec 29 16:09:51 2018 - [info] Replication filtering check ok.
Sat Dec 29 16:09:51 2018 - [info] Starting SSH connection tests..
Sat Dec 29 16:09:58 2018 - [info] All SSH connection tests passed successfully.
Sat Dec 29 16:09:58 2018 - [info] Checking MHA Node version..
Sat Dec 29 16:10:01 2018 - [info] Version check ok.
Sat Dec 29 16:10:01 2018 - [info] Checking SSH publickey authentication settings on the current
master..
Sat Dec 29 16:10:02 2018 - [info] HealthCheck: SSH to server01 is reachable.
Sat Dec 29 16:10:03 2018 - [info] Master MHA Node version is 0.56.
Sat Dec 29 16:10:03 2018 - [info] Checking recovery script configurations on the current
master..
Sat Dec 29 16:10:03 2018 - [info] Executing command: save_binary_logs --command=test
--start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --
manager_version=0.56 --start_file=master-bin.000001 Sat Dec 29 16:10:03 2018 - [info]
Connecting to root@server01(server01)..
Creating /tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to master-bin.000001
Sat Dec 29 16:10:03 2018 - [info] Master setting check done.
Sat Dec 29 16:10:03 2018 - [info] Checking SSH publickey authentication and checking recovery
script configurations on all alive slave servers..
Sat Dec 29 16:10:03 2018 - [info] Executing command : apply_diff_relay_logs
--command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.112
--slave_po
rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56
--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/
--slave_pass=xxxSat Dec 29 16:10:03 2018 - [info] Connecting to
root@192.168.200.112(server02:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Dec 29 16:10:04 2018 - [info] Executing command : apply_diff_relay_logs
--command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.113
--slave_po
rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56
--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/
--slave_pass=xxxSat Dec 29 16:10:04 2018 - [info] Connecting to
root@192.168.200.113(server03:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Dec 29 16:10:05 2018 - [info] Executing command : apply_diff_relay_logs
--command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.114
--slave_po
rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56
--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/
--slave_pass=xxxSat Dec 29 16:10:05 2018 - [info] Connecting to
root@192.168.200.114(server04:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Dec 29 16:10:06 2018 - [info] Slaves settings check done.
Sat Dec 29 16:10:06 2018 - [info]
server01 (current master)
+--server02
+--server03
+--server04
Sat Dec 29 16:10:06 2018 - [info] Checking master_ip_failover_script status:
Sat Dec 29 16:10:06 2018 - [info] /usr/local/bin/master_ip_failover --command=status
--ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m
aster_port=3306
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Checking the Status of the script.. OK
bash: /etc/init.d/keepalived: 没有那个文件或目录
Sat Dec 29 16:10:07 2018 - [info] OK.
Sat Dec 29 16:10:07 2018 - [warning] shutdown_script is not defined.
Sat Dec 29 16:10:07 2018 - [info] Set master ping interval 1 seconds.
Sat Dec 29 16:10:07 2018 - [warning] secondary_check_script is not defined. It is highly
recommended setting it to check master reachability from two or more routes.
Sat Dec 29 16:10:07 2018 - [info] Starting ping health check on
server01(192.168.200.111:3306)..
Thu Aug 31 21:55:23 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't
respond..
注意:其中"Ping(SELECT) succeeded, waiting until MySQL doesn't respond.."说明整个系统已经
开始监控了。
关闭 MHA manager 监控,忽略操作
masterha_stop --conf=/etc/masterha/app1.cnf
发现已经将 VIP:192.168.200.100 绑定在网卡 ens32。
[root@server01 ~]# ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group
default qlen 1000
inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32
inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1
Primary Master(192.168.200.111) 模拟主库故障
[root@server01 ~]# systemctl stop mariadb
[root@server01 ~]# netstat -lnpt | grep :3306
[root@server01 ~]# ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group
default qlen 1000
inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32
slave1(192.168.200.113)状态:
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.112
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000001
Read_Master_Log_Pos: 1372
Relay_Log_File: slave-relay-bin.000002
Relay_Log_Pos: 530
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
slave2(192.168.200.114)状态:
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.112
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000001
Read_Master_Log_Pos: 1372
Relay_Log_File: slave-relay-bin.000002
Relay_Log_Pos: 530
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Server05(192.168.200.115) 监控已经自动关闭:
[root@server05 ~]#
[1]+ 完成 nohup masterha_manager --conf=/etc/masterha/app1.cnf
--remove_dead_master_conf --ignore_last_failover < /dev/null >
/var/log/masterha/app1/manager.log 2>&1
Server05(192.168.200.115) 查看监控配置文件已经发生了变化(server01 的配置已被删除):
[root@server05 ~]# cat /etc/masterha/app1.cnf
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/var/lib/mysql
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=123456
ping_interval=1
remote_workdir=/tmp
repl_password=123456
repl_user=repl
user=root
[server2]
candidate_master=1
check_repl_delay=0
hostname=server02
port=3306
[server3]
hostname=server03
port=3306
[server4]
hostname=server04
port=3306
Server05(192.168.200.115) 故障切换过程中的日志文件内容如下:
[root@server05 ~]# tail -f /var/log/masterha/app1/manager.log
Selected server02 as a new master.
server02: OK: Applying all logs succeeded.
server02: OK: Activated master IP address.
server04: This host has the latest relay log events.
server03: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
server04: OK: Applying all logs succeeded. Slave started, replicating from server02.
server03: OK: Applying all logs succeeded. Slave started, replicating from server02.
server02: Resetting slave info succeeded.
Master failover to server02(192.168.200.112:3306) completed successfully.
故障主库修复及 VIP 切回测试
Primary Master(192.168.200.111):
[root@server01 ~]# systemctl start mariadb
[root@server01 ~]# netstat -lnpt | grep :3306
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
6131/mysqld
Primary Master(192.168.200.111) 指向新的主库
[root@server01 ~]# mysql -u root -p123456
stop slave;
CHANGE MASTER TO
MASTER_HOST='192.168.200.112',
MASTER_USER='repl',
MASTER_PASSWORD='123456';
start slave;
show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.112
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000001
Read_Master_Log_Pos: 1372
Relay_Log_File: mariadb-relay-bin.000002
Relay_Log_Pos: 1208
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Server05(192.168.200.115) 修改监控配置文件添加 server1 配置:
[root@server05 ~]# vim /etc/masterha/app1.cnf
[server01]
hostname=server01
port=3306
Server05(192.168.200.115) 检查集群状态:
[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
-----------------------------------忽略部分信息-----------------------------------
Thu Aug 31 22:20:30 2017 - [info] Alive Servers:
Thu Aug 31 22:20:30 2017 - [info] server01(192.168.200.111:3306)
Thu Aug 31 22:20:30 2017 - [info] server02(192.168.200.112:3306)
Thu Aug 31 22:20:30 2017 - [info] server03(192.168.200.113:3306)
Thu Aug 31 22:20:30 2017 - [info] server04(192.168.200.114:3306)
-----------------------------------忽略部分信息-----------------------------------
server02 (current master)
+--server01
+--server03
+--server04
-----------------------------------忽略部分信息-----------------------------------
MySQL Replication Health is OK.
Server05(192.168.200.115) 开启监控
[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf QQ:981343876 34 / 50 Tel:13651028933
--remove_dead_master_conf --ignore_last_failover<
/dev/null >/var/log/masterha/app1/manager.log 2>&1
&[1] 68199
Secondary Master(192.168.200.112) 关闭现有主库 mysql
[root@server02 ~]# ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group
default qlen 1000
inet 192.168.200.112/24 brd 192.168.200.255 scope global ens32
inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1
[root@server02 ~]# systemctl stop mariadb
[root@server02 ~]# netstat -lnpt | grep :3306
Primary Master(192.168.200.111)
[root@server01 ~]# ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group
default qlen 1000
inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32
inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1
slave1(192.168.200.113)状态:
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.111
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000002
Read_Master_Log_Pos: 923
Relay_Log_File: slave-relay-bin.000002
Relay_Log_Pos: 530
Relay_Master_Log_File: master-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
slave2(192.168.200.114)状态:
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.111
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000002
Read_Master_Log_Pos: 923
Relay_Log_File: slave-relay-bin.000002
Relay_Log_Pos: 530
Relay_Master_Log_File: master-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Server05(192.168.200.115) 配置文件变化(已经移除故障机 server2 配置):
[root@server05 ~]# cat /etc/masterha/app1.cnf
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/var/lib/mysql
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=123456
ping_interval=1
remote_workdir=/tmp
repl_password=123456
repl_user=repl
user=root
[server1]
hostname=server01
port=3306
[server3]
hostname=server03
port=3306
[server4]
hostname=server04
port=3306
Server05(192.168.200.115) 监控日志:
-----------------------------------忽略部分信息-----------------------------------
Selected server01 as a new master.
server01: OK: Applying all logs succeeded.
server01: OK: Activated master IP address.
server03: This host has the latest relay log events.
server04: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
server04: OK: Applying all logs succeeded. Slave started, replicating from server01.
server03: OK: Applying all logs succeeded. Slave started, replicating from server01.
server01: Resetting slave info succeeded.
Master failover to server01(192.168.200.111:3306) completed successfully.
修复 Secondary Master(192.168.200.112)主机:
[root@server02 ~]# systemctl start mariadb
[root@server02 ~]# netstat -lnpt | grep :3306
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
5982/mysqld
Secondary Master(192.168.200.112) 指向新的主库
[root@server02 ~]# mysql -u root -p123456
stop slave;
CHANGE MASTER TO
MASTER_HOST='192.168.200.111',
MASTER_USER='repl',
MASTER_PASSWORD='123456';
start slave;
show slave status\G
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.111
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000002
Read_Master_Log_Pos: 923
Relay_Log_File: mariadb-relay-bin.000003
Relay_Log_Pos: 530
Relay_Master_Log_File: master-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Server05(192.168.200.115)修改监控配置文件添加 server2 配置:
[root@server05 ~]# vim /etc/masterha/app1.cnf
[server2]
hostname=server02
candidate_master=1
port=3306
check_repl_delay=0
Server05(192.168.200.115)检查集群状态:
[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
server01 (current master)
+--server02
+--server03
+--server04
-----------------------------------忽略部分信息-----------------------------------
MySQL Replication Health is OK.
十、配置读请求负载均衡
10.1、安装 LVS 和 Keepalived
192.168.200.116 和 192.168.200.117 主机:
yum -y install ipvsadm kernel-devel openssl-devel keepalived
10.2、master 主机设置
192.168.200.116 lvs master 主机配置:
! Configuration File for keepalived
global_defs {
notification_email {
crushlinux@163.com
}
notification_email_from crushlinux@163.com
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state MASTER
interface ens32
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.200.200
}
}
virtual_server 192.168.200.200 3306 {
delay_loop 6
lb_algo wrr
lb_kind DR
protocol TCP
real_server 192.168.200.113 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 192.168.200.114 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
systemctl restart keepalived
检查 VIP 地址
[root@localhost ~]# ip add | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group
default qlen 1000
inet 192.168.200.116/24 brd 192.168.200.255 scope global ens32
inet 192.168.200.200/32 scope global ens32
查看负载策略
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.200.200:3306 wrr
-> 192.168.200.113:3306 Route 1 0 0
-> 192.168.200.114:3306 Route 1 0 0
10.3、backup 主机设置
192.168.200.40 backup 主机配置:
! Configuration File for keepalived
global_defs {
notification_email {
crushlinux@163.com
}
notification_email_from crushlinux@163.com
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.200.200
}
}
virtual_server 192.168.200.200 3306 {
delay_loop 6
lb_algo wrr
lb_kind DR
protocol TCP
real_server 192.168.200.113 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 192.168.200.114 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
systemctl restart keepalived
10.4、为数据库节点设置 VIP 地址及配置 ARP 相关参数
slave1(192.168.200.113)和 slave2(192.168.200.114)主机配置 realserver 脚本:
[root@server03 ~]# vim realserver.sh
#!/bin/bash
SNS_VIP=192.168.200.200
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
realserver1 检测 VIP 地址
[root@server03 ~]# bash realserver.sh
[root@server03 ~]# ip a | grep lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0
inet 192.168.200.113/24 brd 192.168.200.255 scope global ens32
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
远程复制到 realserver2 主机上
[root@server03 ~]# scp realserver.sh 192.168.200.114:/root/
realserver2 检测 VIP 地址
[root@server04 ~]# bash realserver.sh
[root@server04 ~]# ip a |grep lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0
inet 192.168.200.114/24 brd 192.168.200.255 scope global ens32
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
10.5、数据库读请求测试
Server05(192.168.200.115)担任 client 角色测试,连接 vip
[root@server05 ~]# mysql -uroot -p123456 -h192.168.200.200 -P3306
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 297
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> exit
Bye
断开后在连接测试。。。。
[root@server05 ~]# mysql -uroot -p123456 -h192.168.200.200 -P3306
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 298
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
查看请求分流
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.200.200:3306 wrr
-> 192.168.200.113:3306 Route 1 1 0
-> 192.168.200.114:3306 Route 1 0 1
[root@localhost ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state source virtual destination
TCP 01:48 FIN_WAIT 192.168.200.115:35462 192.168.200.200:3306
192.168.200.114:3306
TCP 14:50 ESTABLISHED 192.168.200.115:35464 192.168.200.200:3306 192.168.200.113:3306
10.6、数据库连接故障处理
[root@localhost ~]# mysql -u root -p123456 -h192.168.200.113 -P3306
ERROR 1129 (HY000): Host '192.168.200.113' is blocked because of many connection errors;
unblock with 'mysqladmin flush-hosts'
[root@server03 ~]# mysqladmin -uroot -p123456 flush-hosts
附录
手动切换 MHA 脚本
[root@server05 ~]# vim /usr/local/bin/master_ip_online_change
#!/usr/bin/env perl
# Copyright (C) 2011 DeNA Co.,Ltd.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
## Note: This is a sample script and is not complete. Modify the script based on your
environment.
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
use Time::HiRes qw( sleep gettimeofday tv_interval );
use Data::Dumper;
my $_tstart;
my $_running_interval = 0.1;
my (
$command, $orig_master_is_new_slave, $orig_master_host,
$orig_master_ip, $orig_master_port, $orig_master_user,
$orig_master_password, $orig_master_ssh_user, $new_master_host,
$new_master_ip, $new_master_port, $new_master_user,
$new_master_password, $new_master_ssh_user
);
my $vip = '192.168.200.100/24'; #VIP
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down";
my $orig_master_ssh_port = 22;
my $new_master_ssh_port = 22;
GetOptions(
'command=s' => \$command,
'orig_master_is_new_slave' => \$orig_master_is_new_slave,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'orig_master_user=s' => \$orig_master_user,
'orig_master_password=s' => \$orig_master_password,
'orig_master_ssh_user=s' => \$orig_master_ssh_user,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
'new_master_password=s' => \$new_master_password,
'new_master_ssh_user=s' => \$new_master_ssh_user,
'orig_master_ssh_port=i' => \$orig_master_ssh_port,
'new_master_ssh_port=i' => \$new_master_ssh_port,
);
exit &main();
sub current_time_us {
my ( $sec, $microsec ) = gettimeofday();
my $curdate = localtime($sec);
return $curdate . "" . sprintf( "%06d", $microsec );
}
sub sleep_until {
my $elapsed =tv_interval($_tstart);
if ( $_running_interval > $elapsed ) {
sleep( $_running_interval - $elapsed );
}
}
sub get_threads_util {
my $dbh = shift;
my $my_connection_id = shift;
my $running_time_threshold = shift;
my $type = shift;
$running_time_threshold = 0 unless ($running_time_threshold);
$type = 0 unless ($type);
my @threads;
my $sth = $dbh->prepare("SHOW PROCESSLIST");
$sth->execute();
while ( my $ref = $sth->fetchrow_hashref() ) {
my $id = $ref->{Id};
my $user = $ref->{User};
my $host = $ref->{Host};
my $command = $ref->{Command};
my $state = $ref->{State};
my $query_time = $ref->{Time};
my $info = $ref->{Info};
$info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
next if ( $my_connection_id == $id );
next if ( defined($query_time) && $query_time < $running_time_threshold );
next if ( defined($command) && $command eq "Binlog Dump" );
next if ( defined($user) && $user eq "system user" );
next
if ( defined($command)
&& $command eq "Sleep"
&& defined($query_time)
&& $query_time >= 1 );
if ( $type >= 1 ) {
next if ( defined($command) && $command eq "Sleep" );
next if ( defined($command) && $command eq "Connect" );
}
if ( $type >= 2 ) {
next if ( defined($info) && $info =~ m/^select/i );
next if ( defined($info) && $info =~ m/^show/i );
}
push @threads, $ref;
}
return @threads;
}
sub main {
if ( $command eq "stop" ) {
## Gracefully killing connections on the current master
# 1. Set read_only= 1 on the new master
# 2. DROP USER so that no app user can establish new connections
# 3. Set read_only= 1 on the current master
# 4. Kill current queries
# * Any database access failure will result in script die.
my $exit_code = 1;
eval {
## Setting read_only=1 on the new master (to avoid accident)
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error(die_on_error)_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
print current_time_us() . " Set read_only on the new master.. ";
$new_master_handler->enable_read_only();
if ( $new_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
$new_master_handler->disconnect();
# Connecting to the orig master, die if any database error happens
my $orig_master_handler = new MHA::DBHelper();
$orig_master_handler->connect( $orig_master_ip, $orig_master_port,
$orig_master_user, $orig_master_password, 1 );
## Drop application user so that nobody can connect. Disabling per-session binlog
beforehand
$orig_master_handler->disable_log_bin_local();
print current_time_us() . " Drpping app user on the orig master..\n";
#FIXME_xxx_drop_app_user($orig_master_handler);
## Waiting for N * 100 milliseconds so that current connections can exit
my $time_until_read_only = 15;
$_tstart = [gettimeofday];
my @threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_read_only > 0 && $#threads >= 0 ) {
if ( $time_until_read_only % 5 == 0 ) {
printf
"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_read_only * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_read_only--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Setting read_only=1 on the current master so that nobody(except SUPER) can write
print current_time_us() . " Set read_only=1 on the orig master.. ";
$orig_master_handler->enable_read_only();
if ( $orig_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
## Waiting for M * 100 milliseconds so that current update queries can complete
my $time_until_kill_threads = 5;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
if ( $time_until_kill_threads % 5 == 0 ) {
printf
"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_kill_threads--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Terminating all threads
print current_time_us() . " Killing all application threads..\n";
$orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );
print current_time_us() . " done.\n";
$orig_master_handler->enable_log_bin_local();
$orig_master_handler->disconnect();
## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
eval {
`ssh -p$orig_master_ssh_port $orig_master_ssh_user\@$orig_master_host \"
$ssh_stop_vip \"`;
};
if ($@) {
warn $@;
}
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
## Activating master ip on the new master
# 1. Create app user with write privileges
# 2. Moving backup script if needed
# 3. Register new master's ip to the catalog database
# We don't return error even though activating updatable accounts/ip failed so that we don't
interrupt slaves' recovery.
# If exit code is 0 or 10, MHA does not abort
my $exit_code = 10;
eval {
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
## Set read_only=0 on the new master
$new_master_handler->disable_log_bin_local();
print current_time_us() . " Set read_only=0 on the new master.\n";
$new_master_handler->disable_read_only();
## Creating an app user on the new master
print current_time_us() . " Creating app user on the new master..\n";
#FIXME_xxx_create_app_user($new_master_handler);
$new_master_handler->enable_log_bin_local();
$new_master_handler->disconnect();
## Update master ip on the catalog database, etc
`ssh -p$new_master_ssh_port $new_master_ssh_user\@$new_master_host \"
$ssh_start_vip \"`;
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
# do nothing
exit 0;
}
else {
&usage();
exit 1;
}
}
sub usage {
print
"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host
--orig_master_ip=ip --orig_master_port=port --new
_master_host=host --new_master_ip=ip --new_master_port=port\n"; die;
}
[root@server05 ~]# chmod +x /usr/local/bin/master_ip_online_change
编辑脚本 send_report
[root@server05 ~]# vim /usr/local/bin/send_report
#!/usr/bin/perl
use strict;
use warnings FATAL => 'all';
use Mail::Sender;
use Getopt::Long;
#new_master_host and new_slave_hosts are set only when recovering master succeeded
my ( $dead_master_host, $new_master_host, $new_slave_hosts, $subject, $body );
my $smtp='smtp.163.com';
my $mail_from='xxxx';
my $mail_user='xxxxx';
my $mail_pass='xxxxx';
my $mail_to=['xxxx','xxxx'];
GetOptions(
'orig_master_host=s' => \$dead_master_host,
'new_master_host=s' => \$new_master_host,
'new_slave_hosts=s' => \$new_slave_hosts,
'subject=s' => \$subject,
'body=s' => \$body,
);
mailToContacts($smtp,$mail_from,$mail_user,$mail_pass,$mail_to,$subject,$body);
sub mailToContacts {
my ( $smtp, $mail_from, $user, $passwd, $mail_to, $subject, $msg ) = @_;
open my $DEBUG, "> /tmp/monitormail.log"
or die "Can't open the debug file:$!\n";
my $sender = new Mail::Sender {
ctype => 'text/plain; charset=utf-8',
encoding => 'utf-8',
smtp => $smtp,
from => $mail_from,
auth => 'LOGIN',
TLS_allowed => '0',
authid => $user,
authpwd => $passwd,
to => $mail_to,
subject => $subject,
debug => $DEBUG
};
$sender->MailMsg(
{ msg => $msg,
debug => $DEBUG
}
) or print $Mail::Sender::Error;
return 1;
}
# Do whatever you want here
exit 0;
[root@server05 ~]# chmod +x /usr/local/bin/send_report
发表于
2019-10-23 09:22
繁梦
阅读( 216)
评论()
编辑
收藏
举报
|
|