(转)主从同步常遇见问题处理-线上MYSQL同步报错故障处理总结
# 在master上更新一条记录,而slave上找不到,丢失了数据
Last_SQL_Error: Could not execute Update_rows_v1 event on table 库名.表名; Can't find record in '库名.表名', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000376, end_log_pos 83010206
/usr/bin/mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS /oa57_data/master_data/mysql-bin.000376 |grep -A '10' 83010206
#200330 11:21:47 server id 10 end_log_pos 83010206 CRC32 0x8e49b876 Update_rows: table id 2766 flags: STMT_END_F ### UPDATE `库名`.`表名` ### WHERE ### @1=1882484 /* INT meta=0 nullable=0 is_null=0 */ ....以下信息省略....
在slave上,查找下更新后的那条记录,应该是不存在的
select * from cwsys.cj_dj where dj_id=1882484\G
在Master上,有数据
其他暂未遇到问题整理:
在master上删除一条记录,而slave上找不到。
Last_SQL_Error: Could not execute Delete_rows event on table hcy.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000006, end_log_pos 254
主键重复。在slave已经有该记录,又在master上插入了同一条记录。
Last_SQL_Error: Could not execute Write_rows event on table hcy.t1; Duplicate entry '2' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin.000006, end_log_pos 924
期间用到的一些小命令:
# 重置密码
set password for 用户名@localhost = password('新密码');
# 查看数据库存放路径
show variables like 'datadir';
DK路径:/oa57_data/master_data/ # 容器路径
真实路径:/data/OA_mysql/data/master_data/ # 物理路径
# 数据权限修订
chown -R 1001:root /data/Test_OA/data/bak
# 数据完整性效验
/usr/bin/docker exec -i mariadb1 innobackupex --defaults=/etc/mysql/my.cnf --apply-log /oa57_data/bak/
# 数据恢复
/usr/bin/docker exec -i mariadb1 innobackupex --defaults-file=/etc/mysql/my.cnf --copy-back /oa57_data/bak/
##################################################################
下面部分存纯COPY,留作以后实操后再进行修订
参考出处:http://xstarcd.github.io/wiki/MySQL/online_mysqlrepl_error.html
##################################################################
异步半同步区别
异步复制
简单的说就是master把binlog发送过去,不管slave是否接收完,也不管是否执行完,这一动作就结束了.
半同步复制
简单的说就是master把binlog发送过去,slave确认接收完,但不管它是否执行完,给master一个信号我这边收到了,这一动作就结束了。(谷歌写的代码,5.5上正式应用。)
异步的劣势
当master上写操作繁忙时,当前POS点例如是10,而slave上IO_THREAD线程接收过来的是3,此时master宕机,会造成相差7个点未传送到slave上而数据丢失。
特殊的情况
- slave的中继日志relay-bin损坏。
Last_SQL_Error: Error initializing relay log position: I/O error reading the header from the binary log Last_SQL_Error: Error initializing relay log position: Binlog has bad magic number; It's not a binary log file that can be used by this version of MySQL
这种情况SLAVE在宕机,或者非法关机,例如电源故障、主板烧了等,造成中继日志损坏,同步停掉。
- 人为失误需谨慎:多台slave存在重复server-id
这种情况同步会一直延时,永远也同步不完,error错误日志里一直出现上面两行信息。解决方法就是把server-id改成不一致即可。
Slave: received end packet from server, apparent master shutdown: Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.000012' at postion 106
问题处理
删除失败
在master上删除一条记录,而slave上找不到。
Last_SQL_Error: Could not execute Delete_rows event on table hcy.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000006, end_log_pos 254
解决方法:
由于master要删除一条记录,而slave上找不到故报错,这种情况主上都将其删除了,那么从机可以直接跳过。可用命令:
stop slave; set global sql_slave_skip_counter=1; start slave;
如果这种情况很多,可用我写的一个脚本skip_error_replcation.sh,默认跳过10个错误(只针对这种情况才跳,其他情况输出错误结果,等待处理),这个脚本是参考maakit工具包的mk-slave-restart原理用shell写的,功能上定义了一些自己的东西,不是无论什么错误都一律跳过。)
主键重复
在slave已经有该记录,又在master上插入了同一条记录。
Last_SQL_Error: Could not execute Write_rows event on table hcy.t1; Duplicate entry '2' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin.000006, end_log_pos 924
解决方法:
在slave上用desc hcy.t1; 先看下表结构:
mysql> desc hcy.t1; +-------+---------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+-----+---------+-------+ | id | int(11) | NO | PRI | 0 | | | name | char(4) | YES | | NULL | | +-------+---------+------+-----+---------+-------+
删除重复的主键
mysql> delete from t1 where id=2; Query OK, 1 row affected (0.00 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G; …… Slave_IO_Running: Yes Slave_SQL_Running: Yes …… mysql> select * from t1 where id=2;
在master上和slave上再分别确认一下。
更新丢失
在master上更新一条记录,而slave上找不到,丢失了数据。
Last_SQL_Error: Could not execute Update_rows event on table hcy.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000010, end_log_pos 794
解决方法:
在master上,用mysqlbinlog 分析下出错的binlog日志在干什么。
/usr/local/mysql/bin/mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS mysql-bin.000010 | grep -A '10' 794 #120302 12:08:36 server id 22 end_log_pos 794 Update_rows: table id 33 flags: STMT_END_F ### UPDATE hcy.t1 ### WHERE ### @1=2 /* INT meta=0 nullable=0 is_null=0 */ ### @2='bbc' /* STRING(4) meta=65028 nullable=1 is_null=0 */ ### SET ### @1=2 /* INT meta=0 nullable=0 is_null=0 */ ### @2='BTV' /* STRING(4) meta=65028 nullable=1 is_null=0 */ # at 794 #120302 12:08:36 server id 22 end_log_pos 821 Xid = 60 COMMIT/*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
在slave上,查找下更新后的那条记录,应该是不存在的。
mysql> select * from t1 where id=2; Empty set (0.00 sec)
然后再到master查看
mysql> select * from t1 where id=2; +----+------+ | id | name | +----+------+ | 2 | BTV | +----+------+ 1 row in set (0.00 sec)
把丢失的数据在slave上填补,然后跳过报错即可。
mysql> insert into t1 values (2,'BTV'); Query OK, 1 row affected (0.00 sec) mysql> select * from t1 where id=2; +----+------+ | id | name | +----+------+ | 2 | BTV | +----+------+ 1 row in set (0.00 sec) mysql> stop slave ;set global sql_slave_skip_counter=1;start slave; Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G; …… Slave_IO_Running: Yes Slave_SQL_Running: Yes ……
中继日志损坏
slave的中继日志relay-bin损坏。
Last_SQL_Error: Error initializing relay log position: I/O error reading the header from the binary log Last_SQL_Error: Error initializing relay log position: Binlog has bad magic number; It's not a binary log file that can be used by this version of MySQL
手工修复
解决方法:找到同步的binlog和POS点,然后重新做同步,这样就可以有新的中继日值了。
例子:
mysql> show slave status\G; *************************** 1. row *************************** Master_Log_File: mysql-bin.000010 Read_Master_Log_Pos: 1191 Relay_Log_File: vm02-relay-bin.000005 Relay_Log_Pos: 253 Relay_Master_Log_File: mysql-bin.000010 Slave_IO_Running: Yes Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 1593 Last_Error: Error initializing relay log position: I/O error reading the header from the binary log Skip_Counter: 1 Exec_Master_Log_Pos: 821 Slave_IO_Running :接收master的binlog信息 Master_Log_File Read_Master_Log_Pos Slave_SQL_Running:执行写操作 Relay_Master_Log_File Exec_Master_Log_Pos
以执行写的binlog和POS点为准。
Relay_Master_Log_File: mysql-bin.000010 Exec_Master_Log_Pos: 821
mysql> stop slave; Query OK, 0 rows affected (0.01 sec) mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000010',MASTER_LOG_POS=821; Query OK, 0 rows affected (0.01 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.8.22 Master_User: repl Master_Port: 3306 Connect_Retry: 10 Master_Log_File: mysql-bin.000010 Read_Master_Log_Pos: 1191 Relay_Log_File: vm02-relay-bin.000002 Relay_Log_Pos: 623 Relay_Master_Log_File: mysql-bin.000010 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1191 Relay_Log_Space: 778 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:
Ibbackup
各种大招都用上了,无奈slave数据丢失过多,ibbackup(需要银子)该你登场了。
Ibbackup热备份工具,是付费的。xtrabackup是免费的,功能上一样。
Ibbackup备份期间不锁表,备份时开启一个事务(相当于做一个快照),然后会记录一个点,之后数据的更改保存在ibbackup_logfile文件里,恢复时把ibbackup_logfile 变化的数据再写入到ibdata里。
Ibbackup 只备份数据( ibdata、.ibd ),表结构.frm不备份。
下面一个演示例子:
备份:ibbackup /bak/etc/my_local.cnf /bak/etc/my_bak.cnf
恢复:ibbackup --apply-log /bak/etc/my_bak.cnf
[root@vm01 etc]# more my_local.cnf datadir =/usr/local/mysql/data innodb_data_home_dir = /usr/local/mysql/data innodb_data_file_path = ibdata1:10M:autoextend innodb_log_group_home_dir = /usr/local/mysql/data innodb_buffer_pool_size = 100M innodb_log_file_size = 5M innodb_log_files_in_group=2 [root@vm01 etc]# ibbackup /bak/etc/my_local.cnf /bak/etc/my_bak.cnf InnoDB Hot Backup version 3.0.0; Copyright 2002-2005 Innobase Oy License A21488 is granted to vm01 (chunyang_he@126.com) (--apply-log works in any computer regardless of the hostname) Licensed for use in a computer whose hostname is 'vm01' Expires 2012-5-1 (year-month-day) at 00:00 See http://www.innodb.com for further information Type ibbackup --license for detailed license terms, --help for help Contents of /bak/etc/my_local.cnf: innodb_data_home_dir got value /usr/local/mysql/data innodb_data_file_path got value ibdata1:10M:autoextend datadir got value /usr/local/mysql/data innodb_log_group_home_dir got value /usr/local/mysql/data innodb_log_files_in_group got value 2 innodb_log_file_size got value 5242880 Contents of /bak/etc/my_bak.cnf: innodb_data_home_dir got value /bak/data innodb_data_file_path got value ibdata1:10M:autoextend datadir got value /bak/data innodb_log_group_home_dir got value /bak/data innodb_log_files_in_group got value 2 innodb_log_file_size got value 5242880 ibbackup: Found checkpoint at lsn 0 1636898 ibbackup: Starting log scan from lsn 0 1636864 120302 16:47:43 ibbackup: Copying log... 120302 16:47:43 ibbackup: Log copied, lsn 0 1636898 ibbackup: We wait 1 second before starting copying the data files... 120302 16:47:44 ibbackup: Copying /usr/local/mysql/data/ibdata1 ibbackup: A copied database page was modified at 0 1636898 ibbackup: Scanned log up to lsn 0 1636898 ibbackup: Was able to parse the log up to lsn 0 1636898 ibbackup: Maximum page number for a log record 0 120302 16:47:46 ibbackup: Full backup completed! [root@vm01 etc]# [root@vm01 etc]# cd /bak/data/ [root@vm01 data]# ls ibbackup_logfile ibdata1 [root@vm01 data]# ibbackup --apply-log /bak/etc/my_bak.cnf InnoDB Hot Backup version 3.0.0; Copyright 2002-2005 Innobase Oy License A21488 is granted to vm01 (chunyang_he@126.com) (--apply-log works in any computer regardless of the hostname) Licensed for use in a computer whose hostname is 'vm01' Expires 2012-5-1 (year-month-day) at 00:00 See http://www.innodb.com for further information Type ibbackup --license for detailed license terms, --help for help Contents of /bak/etc/my_bak.cnf: innodb_data_home_dir got value /bak/data innodb_data_file_path got value ibdata1:10M:autoextend datadir got value /bak/data innodb_log_group_home_dir got value /bak/data innodb_log_files_in_group got value 2 innodb_log_file_size got value 5242880 120302 16:48:38 ibbackup: ibbackup_logfile's creation parameters: ibbackup: start lsn 0 1636864, end lsn 0 1636898, ibbackup: start checkpoint 0 1636898 ibbackup: start checkpoint 0 1636898 InnoDB: Doing recovery: scanned up to log sequence number 0 1636898 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 .....99 Setting log file size to 0 5242880 ibbackup: We were able to parse ibbackup_logfile up to ibbackup: lsn 0 1636898 ibbackup: Last MySQL binlog file position 0 1191, file name ./mysql-bin.000010 ibbackup: The first data file is '/bak/data/ibdata1' ibbackup: and the new created log files are at '/bak/data/' 120302 16:48:38 ibbackup: Full backup prepared for recovery successfully! [root@vm01 data]# ls ibbackup_logfile ibdata1 ib_logfile0 ib_logfile1
把ibdata1 ib_logfile0 ib_logfile1拷贝到从,把.frm也拷贝过去,启动MYSQL后,做同步,那个点就是上面输出的:
ibbackup: Last MySQL binlog file position 0 1191, file name ./mysql-bin.000010
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000010',MASTER_LOG_POS=1191;
Maatkit工具包
- 简介
maatkit是一个开源的工具包,为mysql日常管理提供了帮助。目前,已被Percona公司收购并维护。其中:
mk-table-checksum是用来检测master和slave上的表结构和数据是否一致。
mk-table-sync是发生主从数据不一致时,来修复的。
这两个工具包,没有在现网实际操作的经验,这里仅仅是新技术探讨和学术交流,下面展示下如何使用。
http://www.actionsky.com/products/mysql-others/maatkit.jsp
[root@vm02]# mk-table-checksum h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 -d hcy -t t1 Cannot connect to MySQL because the Perl DBI module is not installed or not found. Run 'perl -MDBI' to see the directories that Perl searches for DBI. If DBI is not installed, try: Debian/Ubuntu apt-get install libdbi-perl RHEL/CentOS yum install perl-DBI OpenSolaris pgk install pkg:/SUNWpmdbi
提示缺少perl-DBI模块,那么直接 yum install perl-DBI。
[root@vm02 bin]# mk-table-checksum h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 -d hcy -t t1 DATABASE TABLE CHUNK HOST ENGINE COUNT CHECKSUM TIME WAIT STAT LAG hcy t1 0 vm02 InnoDB NULL 1957752020 0 0 NULL NULL hcy t1 0 vm01 InnoDB NULL 1957752020 0 0 NULL NULL
如果表数据不一致,CHECKSUM的值是不相等的。
解释下输出的意思:
DATABASE:数据库名 TABLE:表名 CHUNK:checksum时的近似数值 HOST:MYSQL的地址 ENGINE:表引擎 COUNT:表的行数 CHECKSUM:校验值 TIME:所用时间 WAIT:等待时间 STAT:MASTER_POS_WAIT()返回值 LAG:slave的延时时间
如果你想过滤出不相等的都有哪些表,可以用mk-checksum-filter这个工具,只要在后面加个管道符就行了。
[root@vm02 ~]# mk-table-checksum h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 -d hcy | mk-checksum-filter hcy t2 0 vm01 InnoDB NULL 1957752020 0 0 NULL NULL hcy t2 0 vm02 InnoDB NULL 1068689114 0 0 NULL NULL
知道有哪些表不一致,可以用mk-table-sync这个工具来处理。
注:在执行mk-table-checksum时会锁表,表的大小取决于执行的快慢。
MASTER上的t2表数据: SLAVE上的t2表数据: mysql> select * from t2; mysql> select * from t2; +----+------+ +----+------+ | id | name | | id | name | +----+------+ +----+------+ | 1 | a | | 1 | a | | 2 | b | | 2 | b | | 3 | ss | | 3 | ss | | 4 | asd | | 4 | asd | | 5 | ss | +----+------+ +----+------+ 4 rows in set (0.00 sec) 5 rows in set (0.00 sec) mysql> \! hostname; mysql> \! hostname; vm02 vm01
[root@vm02 ~]# mk-table-sync --execute --print --no-check-slave --transaction --databases hcy h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 INSERT INTO `hcy`.`t2`(`id`, `name`) VALUES ('5', 'ss') /*maatkit src_db:hcy src_tbl:t2 src_dsn:h=vm01,p=...,u=admin dst_db:hcy dst_tbl:t2 dst_dsn:h=vm02,p=...,u=admin lock:0 transaction:1 changing_src:0 replicate:0 bidirectional:0 pid:3246 user:root host:vm02*/;
它的工作原理是:先一行一行检查主从库的表是否一样,如果哪里不一样,就执行删除,更新,插入等操作,使其达到一致。表的大小决定着执行的快慢。
If C<--transaction> is specified, C<LOCK TABLES> is not used. Instead, lock and unlock are implemented by beginning and committing transactions. The exception is if L<"--lock"> is 3. If C<--no-transaction> is specified, then C<LOCK TABLES> is used for any value of L<"--lock">. See L<"--[no]transaction">. When enabled, either explicitly or implicitly, the transaction isolation level is set C<REPEATABLE READ> and transactions are started C<WITH CONSISTENT SNAPSHOT>
MySQL复制监控
MySQL常见错误类型
1005:创建表失败 1006:创建数据库失败 1007:数据库已存在,创建数据库失败 1008:数据库不存在,删除数据库失败 1009:不能删除数据库文件导致删除数据库失败 1010:不能删除数据目录导致删除数据库失败 1011:删除数据库文件失败 1012:不能读取系统表中的记录 1020:记录已被其他用户修改 1021:硬盘剩余空间不足,请加大硬盘可用空间 1022:关键字重复,更改记录失败 1023:关闭时发生错误 1024:读文件错误 1025:更改名字时发生错误 1026:写文件错误 1032:记录不存在 1036:数据表是只读的,不能对它进行修改 1037:系统内存不足,请重启数据库或重启服务器 1038:用于排序的内存不足,请增大排序缓冲区 1040:已到达数据库的最大连接数,请加大数据库可用连接数 1041:系统内存不足 1042:无效的主机名 1043:无效连接 1044:当前用户没有访问数据库的权限 1045:不能连接数据库,用户名或密码错误 1048:字段不能为空 1049:数据库不存在 1050:数据表已存在 1051:数据表不存在 1054:字段不存在 1065:无效的SQL语句,SQL语句为空 1081:不能建立Socket连接 1114:数据表已满,不能容纳任何记录 1116:打开的数据表太多 1129:数据库出现异常,请重启数据库 1130:连接数据库失败,没有连接数据库的权限 1133:数据库用户不存在 1141:当前用户无权访问数据库 1142:当前用户无权访问数据表 1143:当前用户无权访问数据表中的字段 1146:数据表不存在 1147:未定义用户对数据表的访问权限 1149:SQL语句语法错误 1158:网络错误,出现读错误,请检查网络连接状况 1159:网络错误,读超时,请检查网络连接状况 1160:网络错误,出现写错误,请检查网络连接状况 1161:网络错误,写超时,请检查网络连接状况 1062:字段值重复,入库失败 1169:字段值重复,更新记录失败 1177:打开数据表失败 1180:提交事务失败 1181:回滚事务失败 1203:当前用户和数据库建立的连接已到达数据库的最大连接数,请增大可用的数据库连接数或重启数据库 1205:加锁超时 1211:当前用户没有创建用户的权限 1216:外键约束检查失败,更新子表记录失败 1217:外键约束检查失败,删除或修改主表记录失败 1226:当前用户使用的资源已超过所允许的资源,请重启数据库或重启服务器 1227:权限不足,您无权进行此操作 1235:MySQL版本过低,不具有本功能
复制监控脚本
参考原文修改。
- 原脚本
1 #!/bin/bash 2 # 3 #check_mysql_slave_replication_status 4 # 5 # 6 # 7 parasum=2 8 help_msg(){ 9 10 cat << help 11 +---------------------+ 12 +Error Cause: 13 +you must input $parasum parameters! 14 +1st : Host_IP 15 +2st : Host_Port 16 help 17 exit 18 } 19 20 [ $# -ne ${parasum} ] && help_msg #若参数不够打印帮助信息并退出 21 22 export HOST_IP=$1 23 export HOST_PORt=$2 24 MYUSER="root" 25 MYPASS="123456" 26 27 MYSQL_CMD="mysql -u$MYUSER -p$MYPASS" 28 MailTitle="" #邮件主题 29 Mail_Address_MysqlStatus="root@localhost.localdomain" #收件人邮箱 30 31 time1=$(date +"%Y%m%d%H%M%S") 32 time2=$(date +"%Y-%m-%d %H:%M:%S") 33 34 SlaveStatusFile=/tmp/salve_status_${HOST_PORT}.${time1} #邮件内容所在文件 35 echo "--------------------Begin at: "$time2 > $SlaveStatusFile 36 echo "" >> $SlaveStatusFile 37 38 #get slave status 39 ${MYSQL_CMD} -e "show slave status\G" >> $SlaveStatusFile #取得salve进程的状态 40 41 #get io_thread_status,sql_thread_status,last_errno 取得以下状态值 42 43 IOStatus=$(cat $SlaveStatusFile|grep Slave_IO_Running|awk '{print $2}') 44 SQLStatus=$(cat $SlaveStatusFile|grep Slave_SQL_Running |awk '{print $2}') 45 Errno=$(cat $SlaveStatusFile|grep Last_Errno | awk '{print $2}') 46 Behind=$(cat $SlaveStatusFile|grep Seconds_Behind_Master | awk '{print $2}') 47 48 echo "" >> $SlaveStatusFile 49 50 if [ "$IOStatus" == "No" ] || [ "$SQLStatus" == "No" ];then #判断错误类型 51 if [ "$Errno" -eq 0 ];then #可能是salve线程未启动 52 $MYSQL_CMD -e "start slave io_thread;start slave sql_thread;" 53 echo "Cause slave threads doesnot's running,trying start slsave io_thread;start slave sql_thread;" >> $SlaveStatusFile 54 MailTitle="[Warning] Slave threads stoped on $HOST_IP $HOST_PORT" 55 elif [ "$Errno" -eq 1007 ] || [ "$Errno" -eq 1053 ] || [ "$Errno" -eq 1062 ] || [ "$Errno" -eq 1213 ] || [ "$Errno" -eq 1032 ]\ 56 || [ "Errno" -eq 1158 ] || [ "$Errno" -eq 1159 ] || [ "$Errno" -eq 1008 ];then #忽略此些错误 57 $MYSQL_CMD -e "stop slave;set global sql_slave_skip_counter=1;start slave;" 58 echo "Cause slave replication catch errors,trying skip counter and restart slave;stop slave ;set global sql_slave_skip_counter=1;slave start;" >> $SlaveStatusFile 59 MailTitle="[Warning] Slave error on $HOST_IP $HOST_PORT! ErrNum: $Errno" 60 else 61 echo "Slave $HOST_IP $HOST_PORT is down!" >> $SlaveStatusFile 62 MailTitle="[ERROR]Slave replication is down on $HOST_IP $HOST_PORT ! ErrNum:$Errno" 63 fi 64 fi 65 if [ -n "$Behind" ];then 66 Behind=0 67 fi 68 echo "$Behind" >> $SlaveStatusFile 69 70 #delay behind master 判断延时时间 71 if [ $Behind -gt 300 ];then 72 echo `date +"%Y-%m%d %H:%M:%S"` "slave is behind master $Bebind seconds!" >> $SlaveStatusFile 73 MailTitle="[Warning]Slave delay $Behind seconds,from $HOST_IP $HOST_PORT" 74 fi 75 76 if [ -n "$MailTitle" ];then #若出错或者延时时间大于300s则发送邮件 77 cat ${SlaveStatusFile} | /bin/mail -s "$MailTitle" $Mail_Address_MysqlStatus 78 fi 79 80 #del tmpfile:SlaveStatusFile 81 > $SlaveStatusFile
- 修改后脚本
- 只做了简单的整理,修正了Behind为NULL的判断,但均未测试;
- 应可考虑增加:
- 对修复执行结果的判断;
- Errno、Behind两种告警分别发邮件?
- 取消SlaveStatusFile临时文件。
- 告警正文增加show slave结果原文。
- 增加PATH,以便加到crontab中。
- 考虑crontab中周期执行、或多个错误的循环修复?
1 #!/bin/sh 2 # check_mysql_slave_replication_status 3 # 参考:http://www.tianfeiyu.com/?p=2062 4 5 Usage(){ 6 echo Usage: 7 echo "$0 HOST PORT USER PASS" 8 } 9 10 [ -z "$1" -o -z "$2" -o -z "$3" -o -z "$4" ] && Usage && exit 1 11 HOST=$1 12 PORT=$2 13 USER=$3 14 PASS=$4 15 16 MYSQL_CMD="mysql -h$HOST -P$PORT -u$USER -p$PASS" 17 18 MailTitle="" #邮件主题 19 Mail_Address_MysqlStatus="root@localhost.localdomain" #收件人邮箱 20 21 time1=$(date +"%Y%m%d%H%M%S") 22 time2=$(date +"%Y-%m-%d %H:%M:%S") 23 24 SlaveStatusFile=/tmp/salve_status_${HOST_PORT}.${time1} #邮件内容所在文件 25 echo "--------------------Begin at: "$time2 > $SlaveStatusFile 26 echo "" >> $SlaveStatusFile 27 28 #get slave status 29 ${MYSQL_CMD} -e "show slave status\G" >> $SlaveStatusFile #取得salve进程的状态 30 31 #get io_thread_status,sql_thread_status,last_errno 取得以下状态值 32 33 IOStatus=$(cat $SlaveStatusFile|grep Slave_IO_Running|awk '{print $2}') 34 SQLStatus=$(cat $SlaveStatusFile|grep Slave_SQL_Running |awk '{print $2}') 35 Errno=$(cat $SlaveStatusFile|grep Last_Errno | awk '{print $2}') 36 Behind=$(cat $SlaveStatusFile|grep Seconds_Behind_Master | awk '{print $2}') 37 38 echo "" >> $SlaveStatusFile 39 40 if [ "$IOStatus" = "No" -o "$SQLStatus" = "No" ];then 41 case "$Errno" in 42 0) 43 # 可能是slave未启动 44 $MYSQL_CMD -e "start slave io_thread;start slave sql_thread;" 45 echo "Cause slave threads doesnot's running,trying start slsave io_thread;start slave sql_thread;" >> $SlaveStatusFile 46 ;; 47 1007|1053|1062|1213|1032|1158|1159|1008) 48 # 忽略这些错误 49 $MYSQL_CMD -e "stop slave;set global sql_slave_skip_counter=1;start slave;" 50 echo "Cause slave replication catch errors,trying skip counter and restart slave;stop slave ;set global sql_slave_skip_counter=1;slave start;" >> $SlaveStatusFile 51 MailTitle="[Warning] Slave error on $HOST:$PORT! ErrNum: $Errno" 52 ;; 53 *) 54 echo "Slave $HOST:$PORT is down!" >> $SlaveStatusFile 55 MailTitle="[ERROR]Slave replication is down on $HOST:$PORT! Errno:$Errno" 56 ;; 57 esac 58 fi 59 60 if [ "$Behind" = "NULL" -o -z "$Behind" ];then 61 Behind=0 62 fi 63 echo "Behind:$Behind" >> $SlaveStatusFile 64 65 #delay behind master 判断延时时间 66 if [ $Behind -gt 300 ];then 67 echo `date +"%Y-%m%d %H:%M:%S"` "slave is behind master $Bebind seconds!" >> $SlaveStatusFile 68 MailTitle="[Warning]Slave delay $Behind seconds,from $HOST $PORT" 69 fi 70 71 if [ -n "$MailTitle" ];then #若出错或者延时时间大于300s则发送邮件 72 cat ${SlaveStatusFile} | /bin/mail -s "$MailTitle" $Mail_Address_MysqlStatus 73 fi 74 75 #del tmpfile:SlaveStatusFile 76 > $SlaveStatusFile
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 没有源码,如何修改代码逻辑?
· 一个奇形怪状的面试题:Bean中的CHM要不要加volatile?
· [.NET]调用本地 Deepseek 模型
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· .NET Core 托管堆内存泄露/CPU异常的常见思路
· 微软正式发布.NET 10 Preview 1:开启下一代开发框架新篇章
· 没有源码,如何修改代码逻辑?
· NetPad:一个.NET开源、跨平台的C#编辑器
· PowerShell开发游戏 · 打蜜蜂
· 凌晨三点救火实录:Java内存泄漏的七个神坑,你至少踩过三个!
2019-03-30 English trip EM2-LP-4B At school Teacher:Will