linux分析dump vmcore文件
- linux分析dump vmcore文件
摘要: 1、安装kernel-debuginfo-common和kernel-debuginfo(下载地址:http://debuginfo.centos.org/6/x86_64/),安装之前,先通过uname-r确认内核版本 [aliyunzixun@xxx.com~]#uname-r2.6.32-358.el6.x86_64 rpm-ivhkernel-debuginfo-common--
1、安装kernel-debuginfo-common 和 kernel-debuginfo(下载地址:http://debuginfo.centos.org/6/x86_64/),安装之前,先通过uname -r 确认内核版本
[aliyunzixun@xxx.com ~]# uname -r
2.6.32-358.el6.x86_64
rpm -ivh kernel-debuginfo-common-x86_64-2.6.32-358.el6.x86_64.rpm
rpm -ivh kernel-debuginfo-2.6.32-358.el6.x86_64.rpm
2、安装crash
yum install crash
3、分析dump(文件名vmcore ),可参考:
http://www.360doc.com/content/16/1012/15/37247901_597864312.shtml
命令操作如下:
打开dump 文件
[aliyunzixun@xxx.com:/tmp/xxxhdpdevxxx34]#crash /usr/lib/debug/lib/modules/2.6.32-431.el6.x86_64/vmlinux /tmp/scchdphiv02134/vmcore
参数解释
/usr/lib/debug/lib/modules/2.6.32-358.el6.x86_64/vmlinux为指定的vmlnux文件,不同版本下不同的vmlinux,
/tmp/scchdphiv02134/vmcore是kdump出来的vmcore文件的路径
主要命令有bt、log、sym ,使用help命令可以查看支持crash工具下支持哪些命令
以上是
- 的内容,更多
- 的内容,请您使用右上方搜索功能获取相关信息。
分类: LINUX
2016-11-25 14:06:44
一,一些依赖包的安装
yum install -y crash
debuginfo-install kernel
二、分析过程
1. crash vmcore /usr/lib/debug/lib/modules/3.10.0-514.el7.x86_64/vmlinux
[16280.734352] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
[16280.735299] IP: [] raid10_sync_request+0x579/0x1a90 [raid10]
[16280.735299] PGD 0
[16280.735299] Oops: 0000 [#1] SMP
[16280.735299] Modules linked in: raid10 ext4 mbcache jbd2 raid1 loop nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec coretemp snd_hda_core iTCO_wdt hp_wmi ppdev sg sparse_keymap gpio_ich iTCO_vendor_support snd_hwdep snd_seq kvm i5000_edac edac_core rfkill snd_seq_device snd_pcm irqbypass pcspkr lpc_ich snd_timer i5k_amb snd parport_pc shpchp parport soundcore dm_multipath nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi nouveau video mxm_wmi drm_kms_helper syscopyarea sysfillrect ahci sysimgblt ata_piix fb_sys_fops libahci ttm libata serio_raw igb drm mptsas dca scsi_transport_sas tg3 i2c_algo_bit mptscsih i2c_core ptp
[16280.735299] mptbase pps_core wmi floppy fjes dm_mirror dm_region_hash dm_log dm_mod
[16280.735299] CPU: 3 PID: 26188 Comm: md0_resync Not tainted 3.10.0-514.el7.x86_64 #1
[16280.735299] Hardware name: Hewlett-Packard HP xw6400 Workstation/0A04h, BIOS 786D4 v02.38 10/25/2010
[16280.735299] task: ffff88003559bec0 ti: ffff8800591e8000 task.ti: ffff8800591e8000
[16280.735299] RIP: 0010:[] [] raid10_sync_request+0x579/0x1a90 [raid10]
[16280.735299] RSP: 0018:ffff8800591ebbc8 EFLAGS: 00010202
[16280.735299] RAX: 00000000002b3780 RBX: ffff88006f8e0b00 RCX: 0000000000000002
[16280.735299] RDX: ffff880053665860 RSI: ffff88006fb58800 RDI: 00000000002b3380
[16280.735299] RBP: ffff8800591ebca0 R08: 0000000000000000 R09: 0000000000000007
[16280.735299] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[16280.735299] R13: ffff880053665840 R14: 0000000000000000 R15: ffff880053fbd400
[16280.735299] FS: 0000000000000000(0000) GS:ffff88007fcc0000(0000) knlGS:0000000000000000
[16280.735299] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[16280.735299] CR2: 00000000000000b8 CR3: 0000000078f19000 CR4: 00000000000007e0
[16280.735299] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[16280.735299] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[16280.735299] Stack:
[16280.735299] ffff8800591ebd14 00000000000003ff 0000000000000000 00000000810d2f4f
[16280.735299] 00000000002b3800 0000000000000000 0000000000000000 ffff880000000007
[16280.735299] ffff88006fb58800 0000000000000000 00000000002b3780 ffff880000000080
[16280.735299] Call Trace:
[16280.735299] [] ? sysfs_notify_dirent+0x50/0x60
[16280.735299] [] md_do_sync+0xabb/0x1000
[16280.735299] [] ? wake_up_atomic_t+0x30/0x30
[16280.735299] [] md_thread+0x155/0x1a0
[16280.735299] [] ? find_pers+0x80/0x80
[16280.735299] [] kthread+0xcf/0xe0
[16280.735299] [] ? end_buffer_async_read+0x130/0x130
[16280.735299] [] ? kthread_create_on_node+0x140/0x140
[16280.735299] [] ret_from_fork+0x58/0x90
[16280.735299] [] ? kthread_create_on_node+0x140/0x140
[16280.735299] Code: ff 31 c0 0f 1f 80 00 00 00 00 48 98 48 c1 e0 05 49 8b 44 05 58 45 89 4d 60 49 89 7d 58 45 89 a5 80 00 00 00 49 89 45 78 4c 8b 33 <49> 8b 96 b8 00 00 00 83 e2 02 0f 84 0f 02 00 00 49 8b 55 68 48
[16280.735299] RIP [] raid10_sync_request+0x579/0x1a90 [raid10]
[16280.735299] RSP
[16280.735299] CR2: 00000000000000b8
2. set 命令
crash> set 26188
PID: 26188
COMMAND: "md0_resync"
TASK: ffff88003559bec0 [THREAD_INFO: ffff8800591e8000]
CPU: 3
STATE: TASK_RUNNING (PANIC)
3. bt命令
crash> bt
PID: 26188 TASK: ffff88003559bec0 CPU: 3 COMMAND: "md0_resync"
#0 [ffff8800591eb868] machine_kexec at ffffffff81059cdb
#1 [ffff8800591eb8c8] __crash_kexec at ffffffff81105182
#2 [ffff8800591eb998] crash_kexec at ffffffff81105270
#3 [ffff8800591eb9b0] oops_end at ffffffff8168ed88
#4 [ffff8800591eb9d8] no_context at ffffffff8167e993
#5 [ffff8800591eba28] __bad_area_nosemaphore at ffffffff8167ea29
#6 [ffff8800591eba70] bad_area_nosemaphore at ffffffff8167eb93
#7 [ffff8800591eba80] __do_page_fault at ffffffff81691b1e
#8 [ffff8800591ebae0] do_page_fault at ffffffff81691cc5
#9 [ffff8800591ebb10] page_fault at ffffffff8168df88
[exception RIP: raid10_sync_request+1401]
RIP: ffffffffa09b09e9 RSP: ffff8800591ebbc8 RFLAGS: 00010202 #RIP 是函数中panic时的地址
RAX: 00000000002b3780 RBX: ffff88006f8e0b00 RCX: 0000000000000002
RDX: ffff880053665860 RSI: ffff88006fb58800 RDI: 00000000002b3380
RBP: ffff8800591ebca0 R8: 0000000000000000 R9: 0000000000000007
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff880053665840 R14: 0000000000000000 R15: ffff880053fbd400
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff8800591ebca8] md_do_sync at ffffffff814fe94b
#11 [ffff8800591ebe50] md_thread at ffffffff814fab65
#12 [ffff8800591ebec8] kthread at ffffffff810b052f
#13 [ffff8800591ebf50] ret_from_fork at ffffffff81696418
4. 反汇编
crash> mod -s raid10
MODULE NAME SIZE OBJECT FILE
ffffffffa09b62e0 raid10 48005 /lib/modules/3.10.0-514.el7.x86_64/kernel/drivers/md/raid10.ko
crash> dis -l raid10_sync_request | grep -C 6 ffffffffa09b09e9
0xffffffffa09b09db : mov %r12d,0x80(%r13)
/usr/src/debug/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.x86_64/drivers/md/raid10.c: 3204
0xffffffffa09b09e2 : mov %rax,0x78(%r13)
/usr/src/debug/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.x86_64/drivers/md/raid10.c: 3206
0xffffffffa09b09e6 : mov (%rbx),%r14
/usr/src/debug/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.x86_64/arch/x86/include/asm/bitops.h: 319
0xffffffffa09b09e9 : mov 0xb8(%r14),%rdx
/usr/src/debug/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.x86_64/drivers/md/raid10.c: 3207
0xffffffffa09b09f0 : and $0x2,%edx
0xffffffffa09b09f3 : je 0xffffffffa09b0c08
/usr/src/debug/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.x86_64/drivers/md/raid10.c: 3220
0xffffffffa09b09f9 : mov 0x68(%r13),%rdx
/usr/src/debug/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.x86_64/drivers/md/raid10.c: 3186
发现是一个位操作,在3207行
3206 rdev = mirror->rdev;
3207 if (!test_bit(In_sync, &rdev->flags)) {
可以知道是mirror->rdev 为空指针。
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 没有源码,如何修改代码逻辑?
· 一个奇形怪状的面试题:Bean中的CHM要不要加volatile?
· [.NET]调用本地 Deepseek 模型
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· .NET Core 托管堆内存泄露/CPU异常的常见思路
· DeepSeek “源神”启动!「GitHub 热点速览」
· 微软正式发布.NET 10 Preview 1:开启下一代开发框架新篇章
· C# 集成 DeepSeek 模型实现 AI 私有化(本地部署与 API 调用教程)
· DeepSeek R1 简明指南:架构、训练、本地部署及硬件要求
· NetPad:一个.NET开源、跨平台的C#编辑器