Java进程占用系统内存较高的排查方法

1、通过top 查看具体是哪个进程占用内存较多

Tasks:  65 total,   1 running,  64 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.0 us,  1.0 sy,  0.0 ni, 96.3 id,  0.3 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  1016168 total,    63544 free,   824060 used,   128564 buff/cache
KiB Swap:  2047996 total,  1612356 free,   435640 used.    49916 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                 
25894 www       20   0 3080304 757036   3252 S  3.3 74.5  25:55.03 java                                                                                                                    
    1 root      20   0   43300   2180   1340 S  0.0  0.2   3:20.35 systemd                                                                                                                 
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.43 kthreadd                                                                                                                
    3 root      20   0       0      0      0 S  0.0  0.0  12:44.14 ksoftirqd/0                                                                                                             
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H                                                                                                            
    7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0       

2、定位线程问题（通过命令查看PID 为25894 进程的线程情况）,命令：# ps p 25894 -L -o pcpu,pmem,pid,tid,time,tname,cmd

# ps p 25894 -L -o pcpu,pmem,pid,tid,time,tname,cmd

%CPU %MEM   PID   TID     TIME TTY      CMD
 0.0 74.4 25894 25894 00:00:00 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.0 74.4 25894 25896 00:00:00 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.1 74.4 25894 25897 00:01:09 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.0 74.4 25894 25898 00:00:00 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.0 74.4 25894 25899 00:00:00 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.0 74.4 25894 25900 00:00:00 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.0 74.4 25894 25901 00:00:00 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.0 74.4 25894 25902 00:00:00 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.1 74.4 25894 25903 00:01:18 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
 0.0 74.4 25894 25904 00:00:14 ?        /App/java/jdk1.8.0_102/bin/java -Djava.util.logging.config.file=/App/tomcat_jtws-opefront/conf/logging.properties -Djava.util.logging.manager=org.ap
.....................................以下部分省略.....................................

# ps p 25894 -L -o pcpu,pmem,pid,tid,time,tname,cmd |wc -l
101

 由此可以看到这个PID:25894的进程产生了很多线程。接下来就可以通过jstack查看内存使用的堆栈，具体可以执行 ps p 25894 -L -o pcpu,pmem,pid,tid,time,tname,cmd |wc -l 来查具体有多少线程. 

3、将PID为25894的堆栈信息打印到jstack.log中，命令: #jstack -l 25894 > jstack.log

4、查看日志，有很多WAITING & TIMED_WAITING 状态的日志信息，具体需要研发查看一下代码的情况；

#cat jstack.log  |grep -E 'WAITING|TIMED_WAITING'
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: WAITING (parking)
   java.lang.Thread.State: WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   java.lang.Thread.State: WAITING (on object monitor)
   java.lang.Thread.State: WAITING (on object monitor)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)
   java.lang.Thread.State: TIMED_WAITING (parking)

后经研发检查代码并升级程序至新版本，问题解决。　　

参考链接：https://www.javatang.com/archives/2017/10/25/36441958.html#TIMED_WAITING