CentOS 7.1系统自动重启的Bug定位过程
【问题】
有同事反应最近有多台MongoDB的服务器CentOS 7.1系统会自动重启，分析了下问题原因。
【排查过程】
1、 检查系统日志/var/log/message，并没有记录异常信息，journalctl相关日志只记录发生过重启
2、 系统默认配置了kdump，使用crash工具分析/var/crash下的转储文件vmcore，命令如下：
crash /usr/lib/debug/lib/modules/3.10.0-327.36.3.el7.x86_64/vmlinux /tmp/vmcore
3、 可以看到kernel crash时的Call Trace，关键信息标注为黄色字体
crash> bt
PID: 9979 TASK: ffff8804b4020b80 CPU: 2 COMMAND: "crond"
#0 [ffff8804b42db778] machine_kexec at ffffffff81051e9b
#1 [ffff8804b42db7d8] crash_kexec at ffffffff810f27e2
#2 [ffff8804b42db8a8] oops_end at ffffffff8163f448
#3 [ffff8804b42db8d0] no_context at ffffffff8162f561
#4 [ffff8804b42db920] __bad_area_nosemaphore at ffffffff8162f5f7
#5 [ffff8804b42db968] bad_area at ffffffff8162f91b
#6 [ffff8804b42db990] __do_page_fault at ffffffff81642235
#7 [ffff8804b42db9f0] trace_do_page_fault at ffffffff81642403
#8 [ffff8804b42dba28] do_async_page_fault at ffffffff81641ae9
#9 [ffff8804b42dba40] async_page_fault at ffffffff8163e678
[exception RIP: netlink_compare+11]
RIP: ffffffff815560bb RSP: ffff8804b42dbaf8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: 000000049f250000 RCX: 00000000c3637c42
RDX: 00000000000026fb RSI: ffff8804b42dbb48 RDI: 000000049f24fb78
RBP: ffff8804b42dbb30 R8: ffff8804b42dbb44 R9: 0000000000002170
R10: 0000000000000000 R11: ffff8804b42db966 R12: ffff88061dcd2678
R13: ffff8804b42dbb48 R14: ffffffff815560b0 R15: ffff88061b639000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff8804b42dbb00] rhashtable_lookup_compare at ffffffff813080d0
#11 [ffff8804b42dbb38] netlink_lookup at ffffffff815569ee
#12 [ffff8804b42dbb68] netlink_getsockbyportid at ffffffff81557d8f
#13 [ffff8804b42dbb80] netlink_alloc_skb at ffffffff81557dff
#14 [ffff8804b42dbbb8] netlink_ack at ffffffff8155a8a9
#15 [ffff8804b42dbbf0] audit_receive at ffffffff811067e7
#16 [ffff8804b42dbc18] netlink_unicast at ffffffff8155a02d
#17 [ffff8804b42dbc60] netlink_sendmsg at ffffffff8155a420
#18 [ffff8804b42dbcf8] sock_sendmsg at ffffffff815112d0
#19 [ffff8804b42dbe58] SYSC_sendto at ffffffff81511841
#20 [ffff8804b42dbf70] sys_sendto at ffffffff815122ce
#21 [ffff8804b42dbf80] system_call_fastpath at ffffffff81646b49
RIP: 00007f4ac19d5353 RSP: 00007ffe233b1fb8 RFLAGS: 00010202
RAX: 000000000000002c RBX: ffffffff81646b49 RCX: 0000000000000000
RDX: 000000000000009c RSI: 00007ffe233b1ff0 RDI: 0000000000000003
RBP: 00007ffe233b1ff0 R8: 00007ffe233b1fe0 R9: 000000000000000c
R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff815122ce
R13: ffff8804b42dbf78 R14: 000000000000044d R15: 0000000000000001
ORIG_RAX: 000000000000002c CS: 0033 SS: 002b
4、在网上搜索，定位到这是kernel Linux 3.10.0-327.36.3.el7.x86_64的bug，详细描述可以参见下面，该bug在 7.3 kernel (3.10.0-514.el7)后修复
https://bugs.centos.org/view.php?id=12012
5、 但考虑到升级系统成本较高，后面尝试定位触发条件，可以看到触发这个bug是crond命令
PID: 9979 TASK: ffff8804b4020b80 CPU: 2 COMMAND: "crond"
6、 借助systemtap工具，在发生crash的kernel函数上加探针，kernel.function("rhashtable_lookup_compare")
抓取到crond等系统命令确实会调用上面的函数
7、考虑到Mongo最近新上了一套系统监控的脚本是通过crontab调度的，而上监控之前服务器重启的情况很少，猜测可能是crontab调度系统监控采集触发了kernel bug
后面计划将采集脚本改为服务的方式观察是否可以规避触发bug
【结论】
由于已发现当前CentOS 7.1存在自动重启的bug，建议后面新上的linux服务器都采用CentOS7.4的系统
Description: CentOS Linux release 7.1.1503 (Core)
Release: 7.1.1503
Linux 3.10.0-327.36.3.el7.x86_64
数据库运维经验分享&MySQL源码学习漫漫路
