又见一帘 kernel oops
1 [866039.417492] qemu-kvm invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 2 [866039.419212] qemu-kvm cpuset=vcpu0 mems_allowed=0 3 [866039.420934] CPU: 3 PID: 15224 Comm: qemu-kvm Tainted: G E ------------ 3.10.0-1127.el7.x86_64 #1 4 [866039.422744] Hardware name: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 5.6.5 09/14/2018 5 [866039.424541] Call Trace: 6 [866039.426313] [<ffffffff88d7ff85>] dump_stack+0x19/0x1b 7 [866039.428089] [<ffffffff88d7a8a3>] dump_header+0x90/0x229 8 [866039.429902] [<ffffffffc049a965>] ? i915_gem_shrink_all+0x35/0x40 [i915] 9 [866039.431699] [<ffffffffc049aa6a>] ? i915_gem_shrinker_oom+0xfa/0x180 [i915] 10 [866039.433434] [<ffffffff887c246e>] oom_kill_process+0x25e/0x3f0 11 [866039.435164] [<ffffffff88733a41>] ? cpuset_mems_allowed_intersects+0x21/0x30 12 [866039.436915] [<ffffffff887c1ecd>] ? oom_unkillable_task+0xcd/0x120 13 [866039.438654] [<ffffffff887c1f76>] ? find_lock_task_mm+0x56/0xc0 14 [866039.440388] [<ffffffff887c2cc6>] out_of_memory+0x4b6/0x4f0 15 [866039.442124] [<ffffffff88d7b3c0>] __alloc_pages_slowpath+0x5db/0x729 16 [866039.443848] [<ffffffff887c9146>] __alloc_pages_nodemask+0x436/0x450 17 [866039.445542] [<ffffffff88818e18>] alloc_pages_current+0x98/0x110 18 [866039.447277] [<ffffffff887be377>] __page_cache_alloc+0x97/0xb0 19 [866039.448977] [<ffffffff887c0f30>] filemap_fault+0x270/0x420 20 [866039.450698] [<ffffffffc06bea4e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] 21 [866039.452429] [<ffffffffc06bec4c>] xfs_filemap_fault+0x2c/0x30 [xfs] 22 [866039.454145] [<ffffffff887edeea>] __do_fault.isra.61+0x8a/0x100 23 [866039.455866] [<ffffffff887ee49c>] do_read_fault.isra.63+0x4c/0x1b0 24 [866039.457580] [<ffffffff887f5d00>] handle_mm_fault+0xa20/0xfb0 25 [866039.459298] [<ffffffff88d8d653>] __do_page_fault+0x213/0x500 26 [866039.460985] [<ffffffff88d8d975>] do_page_fault+0x35/0x90 27 [866039.462685] [<ffffffff88d89778>] page_fault+0x28/0x30 28 [866039.464360] Mem-Info: 29 [866039.466000] active_anon:591382 inactive_anon:33937 isolated_anon:0 30 active_file:0 inactive_file:94 isolated_file:20 31 unevictable:0 dirty:0 writeback:0 unstable:0 32 slab_reclaimable:24764 slab_unreclaimable:17585 33 mapped:5313 shmem:76166 pagetables:5132 bounce:0 34 free:26118 free_pcp:222 free_cma:0 35 [866039.476226] Node 0 DMA free:15900kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 36 [866039.482100] lowmem_reserve[]: 0 2864 7855 7855 37 [866039.484206] Node 0 DMA32 free:45832kB min:24596kB low:30744kB high:36892kB active_anon:756268kB inactive_anon:39556kB active_file:24kB inactive_file:428kB unevictable:0kB isolated(anon):0kB isolated(file):80kB present:3018104kB managed:2936540kB mlocked:0kB dirty:0kB writeback:0kB mapped:5864kB shmem:90864kB slab_reclaimable:30792kB slab_unreclaimable:21044kB kernel_stack:2208kB pagetables:6652kB unstable:0kB bounce:0kB free_pcp:148kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 38 [866039.493388] lowmem_reserve[]: 0 0 4990 4990 39 [866039.495805] Node 0 Normal free:42740kB min:42848kB low:53560kB high:64272kB active_anon:1609260kB inactive_anon:96192kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5242880kB managed:5110280kB mlocked:0kB dirty:0kB writeback:0kB mapped:15388kB shmem:213800kB slab_reclaimable:68264kB slab_unreclaimable:49296kB kernel_stack:3712kB pagetables:13876kB unstable:0kB bounce:0kB free_pcp:744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:97 all_unreclaimable? yes 40 [866039.506365] lowmem_reserve[]: 0 0 0 0 41 [866039.509111] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB 42 [866039.512048] Node 0 DMA32: 195*4kB (UEM) 224*8kB (UEM) 712*16kB (UEM) 471*32kB (UEM) 164*64kB (UEM) 42*128kB (UEM) 2*256kB (UE) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45420kB 43 [866039.517816] Node 0 Normal: 325*4kB (UEM) 297*8kB (UE) 707*16kB (UE) 487*32kB (UE) 172*64kB (UEM) 10*128kB (EM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42860kB 44 [866039.523835] Node 0 hugepages_total=2518 hugepages_free=22 hugepages_surp=0 hugepages_size=2048kB 45 [866039.526781] 76305 total pagecache pages 46 [866039.529714] 0 pages in swap cache 47 [866039.532641] Swap cache stats: add 0, delete 0, find 0/0 48 [866039.535579] Free swap = 0kB 49 [866039.538521] Total swap = 0kB 50 [866039.541426] 2069242 pages RAM 51 [866039.544324] 0 pages HighMem/MovableOnly 52 [866039.547204] 53562 pages reserved 53 [866039.550072] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 54 [866039.553016] [ 437] 0 437 11250 153 24 0 -1000 systemd-udevd 55 [866039.555979] [ 551] 81 551 14559 159 33 0 -900 dbus-daemon 56 [866039.558949] [ 565] 0 565 13230 217 30 0 0 smartd 57 [866039.561912] [ 567] 0 567 6595 80 17 0 0 systemd-logind 58 [866039.564877] [ 575] 0 575 1097 34 7 0 0 acpid 59 [866039.567829] [ 616] 0 616 31595 154 18 0 0 crond 60 [866039.570770] [ 1022] 0 1022 28231 257 58 0 -1000 sshd 61 [866039.573704] [ 1767] 0 1767 1088 25 8 0 0 init_bond 62 [866039.576638] [ 1784] 0 1784 13322 143 28 0 0 vsftpd 63 [866039.579561] [ 1793] 0 1793 3262 69 10 0 0 time_server 64 [866039.582487] [ 1814] 0 1814 27551 34 10 0 0 agetty 65 [866039.585412] [ 2589] 0 2589 401290 1979 169 0 0 libvirtd 66 [866039.588343] [ 4617] 0 4617 56533 262 66 0 -900 virtlogd 67 [866039.591277] [ 4992] 0 4992 1086 24 8 0 0 arp_proxy 68 [866039.594205] [ 6077] 0 6077 13507 167 32 0 0 socat 69 [866039.597123] [ 5623] 0 5623 13507 165 29 0 0 socat 70 [866039.600013] [ 9462] 0 9462 63878 4314 59 0 0 rsyslogd 71 [866039.602907] [ 9470] 0 9470 17963 7881 41 0 0 systemd-journal 72 [866039.605800] [ 9493] 0 9493 1089 29 8 0 0 fs_monitor 73 [866039.608679] [ 9499] 0 9499 164592 199 41 0 0 kvm_vswitch 74 [866039.611550] [ 9507] 0 9507 644552 1081 106 0 0 fs_agent 75 [866039.614344] [ 9513] 0 9513 40671 164 15 0 0 host_resource 76 [866039.617059] [ 9525] 0 9525 42725 99 16 0 0 kvm_bash 77 [866039.619692] [ 9539] 0 9539 3263 75 9 0 0 time_client 78 [866039.622244] [ 9556] 0 9556 79698 2332 23 0 0 host_monitor 79 [866039.624723] [10339] 0 10339 117559 510 31 0 0 fuse_client 80 [866039.627122] [10343] 0 10343 1092 23 8 0 0 fusermount 81 [866039.629439] [14851] 107 14851 426229 14799 249 0 0 qemu-kvm 82 [866039.631674] [14852] 0 14852 5509 57 18 0 0 systemd-machine 83 [866039.633913] [15216] 107 15216 397757 8540 193 0 0 qemu-kvm 84 [866039.636088] [20096] 107 20096 397693 8148 190 0 0 qemu-kvm 85 [866039.638170] [22448] 107 22448 397692 7638 193 0 0 qemu-kvm 86 [866039.640169] [29173] 0 29173 5096 147 13 0 0 nginx 87 [866039.642080] [29174] 99 29174 5708 876 15 0 0 nginx 88 [866039.643972] [12084] 0 12084 39731 332 79 0 0 sshd 89 [866039.645790] [12088] 0 12088 28919 134 15 0 0 bash 90 [866039.647533] [12215] 0 12215 39731 352 79 0 0 sshd 91 [866039.649190] [12219] 0 12219 28919 126 14 0 0 bash 92 [866039.650760] [12256] 0 12256 28320 50 12 0 0 pc 93 [866039.652236] [12257] 0 12257 45792 246 45 0 0 ssh 94 [866039.653665] [14223] 0 14223 39764 364 79 0 0 sshd 95 [866039.655057] [14228] 0 14228 28919 129 14 0 0 bash 96 [866039.656449] [24577] 0 24577 5060 115 15 0 0 nginx 97 [866039.657832] [24578] 99 24578 5171 263 16 0 0 nginx 98 [866039.659196] [26810] 0 26810 39731 358 79 0 0 sshd 99 [866039.660570] [26816] 0 26816 18063 189 38 0 0 sftp-server 100 [866039.661966] [27470] 0 27470 28320 50 13 0 0 pc 101 [866039.663358] [27471] 0 27471 45792 254 45 0 0 ssh 102 [866039.664743] [21994] 999 21994 153087 873 62 0 0 polkitd 103 [866039.666108] [26135] 0 26135 39732 332 81 0 0 sshd 104 [866039.667446] [26166] 0 26166 28920 129 14 0 0 bash 105 [866039.668770] [29097] 99 29097 13507 169 31 0 0 socat 106 [866039.670095] [ 4887] 0 4887 32138 84 20 0 0 ping 107 [866039.671416] [ 4919] 0 4919 32138 83 22 0 0 ping 108 [866039.672737] [13221] 0 13221 39732 344 80 0 0 sshd 109 [866039.674057] [13226] 0 13226 28919 136 14 0 0 bash 110 [866039.675365] [14622] 0 14622 31210 86 27 0 0 lvmetad 111 [866039.676671] [15186] 0 15186 39732 333 79 0 0 sshd 112 [866039.677975] [15199] 0 15199 28919 127 14 0 0 bash 113 [866039.679274] [20722] 107 20722 491100 9624 215 0 0 qemu-kvm 114 [866039.680585] [22605] 0 22605 39732 341 81 0 0 sshd 115 [866039.681901] [22611] 0 22611 28919 131 14 0 0 bash 116 [866039.683207] [22645] 0 22645 39732 333 80 0 0 sshd 117 [866039.684497] [22649] 0 22649 28919 133 13 0 0 bash 118 [866039.685782] [27447] 107 27447 488595 11851 213 0 0 qemu-kvm 119 [866039.687079] [27927] 107 27927 504418 283629 757 0 0 qemu-kvm 120 [866039.688375] [28386] 107 28386 504683 273496 758 0 0 qemu-kvm 121 [866039.689659] [29819] 0 29819 28320 43 13 0 0 sh 122 [866039.690943] [29820] 0 29820 42438 272 78 0 0 virsh 123 [866039.692223] [29821] 0 29821 28202 33 14 0 0 grep 124 [866039.693498] [29945] 0 29945 45615 229 45 0 0 crond 125 [866039.694766] [29946] 0 29946 45615 229 45 0 0 crond 126 [866039.696017] [30021] 0 30021 28313 40 14 0 0 sh 127 [866039.697265] Out of memory: Kill process 27927 (qemu-kvm) score 141 or sacrifice child 128 [866039.698544] Killed process 27927 (qemu-kvm), UID 107, total-vm:2017672kB, anon-rss:1134500kB, file-rss:0kB, shmem-rss:16kB