关于used下的内存使用情况分析
1、来由
开发同事反应本机没有做什么，内存就占用了11G，也不知道是什么进程占用了，为此我展开了used内存占用的分析，上数据图
# free -m
total used free shared buffers cached
Mem: 32111 11316 20794 0 157 1396
-/+ buffers/cache: 9762 22348
Swap: 8047 64 7983
2、原因分析
针对此问题，我及时查看哪个进程占用
# ps aux | head -1; ps aux| sort -rnk 4 | head -10
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
zabbix 22866 0.0 0.0 78244 1836 ? S Jan18 0:27 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
zabbix 22865 0.0 0.0 78244 1856 ? S Jan18 0:27 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
zabbix 22864 0.0 0.0 78244 1852 ? S Jan18 0:27 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
zabbix 22863 0.0 0.0 78120 1924 ? S Jan18 0:59 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
zabbix 22861 0.0 0.0 78120 768 ? S Jan18 0:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
rtkit 2855 0.0 0.0 168452 844 ? SNl 2020 4:55 /usr/libexec/rtkit-daemon
rpcuser 2212 0.0 0.0 23344 740 ? Ss 2020 0:00 rpc.statd
rpc 2092 0.0 0.0 18972 604 ? Ss 2020 0:16 rpcbind
root 99 0.0 0.0 0 0 ? S 2020 0:00 [kintegrityd/9]
但是没有找到问题点，继续
查看/proc/meminfo文件时
# cat /proc/meminfo 未启动java时 启动java后
MemTotal: 32882392 kB 32882392 kB
MemFree: 11152484 kB 9941460 kB
Buffers: 174520 kB 175588 kB
Cached: 1485864 kB 1508656 kB
SwapCached: 5100 kB 5092 kB
Active: 10906000 kB 12098640 kB
Inactive: 845964 kB 859936 kB
Active(anon): 10088872 kB 11271860 kB
Inactive(anon): 4448 kB 4448 kB
Active(file): 817128 kB 826780 kB
Inactive(file): 841516 kB 855488 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8241144 kB 8241144 kB
SwapFree: 8174820 kB 8174828 kB
Dirty: 980 kB 1472 kB
Writeback: 0 kB
AnonPages: 10090488 kB 11275588 kB
Mapped: 35960 kB 35964 kB
Shmem: 1600 kB 1600 kB
Slab: 9708836 kB 9709656 kB
SReclaimable: 8982448 kB 8982488 kB
SUnreclaim: 726388 kB 727168 kB
KernelStack: 5592 kB 5888 kB
PageTables: 34344 kB 36660 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24682340 kB 24682340 kB
Committed_AS: 21549588 kB 21577332 kB
VmallocTotal: 34359738367 kB 34359738367 kB
VmallocUsed: 200220 kB 200220 kB
VmallocChunk: 34341972664 kB 34341972664 kB
HardwareCorrupted: 0 kB
AnonHugePages: 9539584 kB 10670080 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB 2048 kB
DirectMap4k: 10240 kB 10240 kB
DirectMap2M: 33544192 kB 33544192 kB
3、在网络上找原因
自己分析不出问题点，网络上有很多人说slab来分配内存，但是我想知道slab中内存包括那些内容，很少的知识点，怎么办？
我找到一个工具，linux-fincore
下载linux-fincore工具：
#wget https://codeload.github.com/waleedmazhar/linux-ftools/zip/master
安装步骤如下：
#cd /opt
#unzip -d linux-ftools-master.zip
#cd linux-ftools-master
#./configure --prefix=/usr/local/linux-ftools #请确保安装gcc依赖包，无错误进行下一步
#make
#make install
添加环境变量：
#mv /usr/local/ftools/bin/* /usr/local/bin/
添加命令使用
`# linux-fincore --pages=false --summarize --only-cached *
filename size total_pages min_cached page cached_pages cached_size cached_perc
-------- ---- ----------- --------------- ------------ ----------- -----------
Could not mmap file: linux-ftools-master: No such device
netapp.txt 891,161 218 0 218 892,928 100.00
---
total cached size: 892,928 `
为了更好抓取内存中进程，我这边添加个脚本+定时任务来完成，提供开发需要的资料，
脚本为：
# cat fincore.sh
#!/bin/bash
. /etc/init.d/functions
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/lib:/root/bin
echo $PATH
if [ ! -f /usr/local/bin/linux-fincore ];then
echo "you haven't install linux-fincore."
exit 1
fi
ps -e -o pid,rss|sort -rnk 2 | head -10 | awk '{print $1}' > /tmp/cache.pid
if [ -f /tmp/cache.log ];then
echo "the cache.files is exist,removing now"
rm -f /tmp/cache.log
fi
while read line
do
lsof -p $line 2>/dev/null | awk '{print $9}' >>/tmp/cache.log
done > /tmp/cache.fincore
fi
done
linux-fincore -s `cat /tmp/cache.fincore`
rm -f /tmp/cache.{pid,log,fincore}
测试一下脚本使用
#/bin/bash fincore.sh > 2.log
#查看日志中分析的数据
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/java/jdk1.8.0_131/bin:/usr/java/jdk1.8.0_131/lib:/root/bin
filename size total_pages min_cached page cached_pages cached_size cached_perc
-------- ---- ----------- --------------- ------------ ----------- -----------
/weaver/jdk1.8.0_131/bin/java 7,734 2 0 1 4,096 50.00
/lib64/libuuid.so.1.3.0 18,936 5 0 5 20,480 100.00
/usr/lib64/libstdc++.so.6.0.13 989,840 242 0 109 446,464 45.04
/usr/lib64/libsapjco3.so 5,407,699 1,321 0 500 2,048,000 37.85
/weaver/ecology/WEB-INF/lib/ehcache-2.8.3.jar 5,637,564 1,377 0 454 1,859,584 32.97
这边我只保留部分数据，
这时，脚本已启作用了，现在只需要添加定时任务让它自动运行并输出日志给到系统了，方便他们及时查看。
