「Bug」CentOS7 报 OOM,systemd 内存占用率达到 57.8%
top 命令输出(输入 M 进程按内存排序,输入 m 可视化内存占用):
top - 13:24:35 up 22 days, 23:01, 1 user, load average: 15.64, 18.52, 12.97
Tasks: 358 total, 5 running, 353 sleeping, 0 stopped, 0 zombie
%Cpu(s): 10.3 us, 21.7 sy, 0.0 ni, 61.5 id, 1.5 wa, 0.0 hi, 5.1 si, 0.0 st
KiB Mem : 98.5/16247608 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
KiB Swap: 0.0/0 [ ]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 9605472 9.0g 348 S 0.0 57.8 798:26.87 systemd
82720 polkitd 20 0 4978420 276312 0 S 2.3 1.7 29:45.70 mysqld
57250 root 20 0 6290796 193804 11516 S 43.9 1.2 4573:23 kubelet
97595 root 20 0 256856 166472 44 R 30.8 1.0 235:49.03 ruby
57071 root 20 0 1796564 110960 564 S 15.1 0.7 2129:00 dockerd
82641 root 20 0 26.1g 85152 0 S 2.0 0.5 37:26.06 dotnet
82103 root 20 0 26.1g 81612 764 S 3.3 0.5 75:23.47 dotnet
81469 root 20 0 26.1g 78808 852 S 2.6 0.5 37:30.94 dotnet
85327 root 20 0 26.1g 78404 0 S 0.3 0.5 15:03.34 dotnet
15729 polkitd 20 0 1853084 78152 1876 S 2.6 0.5 8:14.68 mongod
89447 root 20 0 26.1g 77728 0 S 3.6 0.5 36:48.23 dotnet
systemctl status
发现状态为 degraded
systemctl --failed
输出:
UNIT LOAD ACTIVE SUB DESCRIPTION
● chronyd.service loaded failed failed NTP client/server
● irqbalance.service loaded failed failed irqbalance daemon
● postfix.service loaded failed failed Postfix Mail Transport Agent
● tuned.service loaded failed failed Dynamic System Tuning Daemon
● vgauthd.service loaded failed failed VGAuth Service for open-vm-tools
● vmtoolsd.service loaded failed failed Service for virtual machines hosted on VMware
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
6 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
感觉上很像 systemd 发生了内存泄漏,但是 journalctl
的日志看不明白,ps 1
(1 即 systemd pid) 搞一圈也找不到问题点。。。
网上搜一圈有说是 CentOS7 的 bug。给出的解决方案,我总结如下:
- 重启虚拟机,回收内存。
- 实测网上给出的回收内存命令:
systemctl daemon-reexec
没有任何效果。。 - 只能关闭电源,实测
reboot
/poweroff
/init 0
命令均无法关机,报错:
- 实测网上给出的回收内存命令:
Failed to start poweroff.target: Connection timed out
See system logs and 'systemctl status poweroff.target' for details.
Broadcast message from root@xxx-xxx on pts/0 (Tue 2020-11-24 13:45:11 CST):
The system is going down for power-off NOW!
- 升级 systemd 到新版本,新版本应该修复了这个 Bug:
yum update systemd
- 或者升级整个系统
yum update
- 或者升级整个系统
升级后应该就没毛病了.