运维故障一之mysql宕掉

运维故障一之mysql宕掉

 

曾出现过两次：

原因一：请参考《linux系统硬件一之磁盘逻辑坏道修复》

原因二：

1.查看mysql日志，报错找不到 slowquery.log，但该文件是存在的，排除；若真不存在，可以手动创建并赋予mysql权限

InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
170910 20:40:32  InnoDB: Started; log sequence number 0 44233
/usr/libexec/mysqld: File './data/slowquery.log' not found (Errcode: 2)
170910 20:40:32 [ERROR] Could not use ./data/slowquery.log for logging (error 2). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
170910 20:40:32 [Note] Recovering after a crash using mysql-bin
170910 20:40:37 [Note] Starting crash recovery...

2.查看系统日志dmesg，提示内存不够，杀死了mysql

Out of memory: Kill process 29057 (mysqld) score 21 or sacrifice child
Killed process 29057, UID 27, (mysqld) total-vm:9770172kB, anon-rss:664248kB, file-rss:200kB
swapper: page allocation failure. order:1, mode:0x20  #内核问题，建议升级系统内核，临时设置参数可以解决：vm.zone_reclaim_mode = 1  当内存不够用时，系统回收buff/cache
Pid: 0, comm: swapper Not tainted 2.6.32-279.el6.x86_64 #1

3.查询什么进程占用了大量的内存，最后发现时一个python脚本运行7000多次，经查看是脚本执行有问题导致进程挂起消耗了内存

4.解决内存问题，杀死所有的异常的python脚本进程

for i in `ps aux | grep disk_used_info.py | awk '{print $2}'`;do kill $i;done

5.完美解决，可以给运营答复了.............

6.异常python脚本问题待确定.