Java本地内存泄露分析 Java – Debugging Native Memory Leaks
When we think about memory leaks in Java, we usually do it for Java heap memory leak which is used to store objects and are if they are not garbage collected ever. But there can be scenarios where native memory in the non-heap memory starts leaking. . The objective of this blog post is to discuss native memory leaks, and how we can detect and troubleshoot them.
Java Memory Model
Java memory pools that application has access to and can allocate from. We all know, java objects are allocated into the Java Heap. Besides the Java Heap, there are other JVM managed memory spaces such as Metaspace, Stackspace, Native Area and Reserved . In addition, the JVM and the native code of applications can make allocations out of native memory. These native allocation can be done by JNI allocations, allocations made by the JIT compiler, or creation of new threads by the JVM. As native allocations can be made from several different places out of your code, it is not easy to debug them.
Symptoms
- In normal case sum of heap and non-heap memory consumed by JVM should be comparable to the total memory consumption (RSS) of the java application. But if RSS is constantly increasing with no increase in heap and non-heap memory, then you have a native memory leak. RSS though is not a very reliable way as it has some issue tracking exact memory consumed by application use PSS instead. It is basically the sum physical and shared memory per mapper.
- Server crashing by Linux out of memory killer which is OS mechanism that frees memory when the system gets to a stressful low memory situation.
java.lang.OutOfMemoryError: requested 1234bytes for ChunkPool::allocate. Out of swap space? Internal Error (allocation.cpp:166), pid=1190, tid=20# Error: ChunkPool::allocate
java.lang.OutOfMemoryError : unable to create new native Thread
You may find these two errors in logs which will tell you that your JVM has run out of native memory. This will happen when you have restricted the max memory of your application or the host itself ran out of memory.
Reasons
- Any application calls malloc() to acquire memory. If it discards the reference without calling free(), the OS thinks process has memory but it has already lost the reference to it.
- Could be due to JNI allocations, JIT compiler allocations or creation of threads
- Class loader leak (java.lang.Class instance could not be garbage collected after deploy) due to this metaspace increases and resulting in increase of RSS.
- Leaking third party code.
Tools we will be using
- The jcmd Utility : Consists of NMT + Thread/GC/VM monitoring tools + JFR
- Native Memory Tracking: Tracking native memory leak in JVM
- HPROF : For heap profiling
- Java VisualVM: Memory consumption, running threads, classes monitoring.
- The jdb Utility : For connectors and listeners
- The jmap Utility: Heap dump
- The jstack Utility : Thread dump
- gdb: The GNU Project Debugger
- Core dump analysis
Searching for the leak
1. Confirm a native memory leak
First monitor your application JVM heap and non-heap memory using jconsole and watch -n 1 ps v to monitor rss of the application. Check if PSS for your application is growing over time, calculate it by adding the output for the below command:
cat /proc/
When you can confirm growth of PSS over time and heap more or less remaining constant, this confirms a leak in native memory. As mentioned above every application calls malloc/mmap to acquire memory, and free to give it back. So to search for memory leak you need to monitor these calls. There are some tools like jemalloc, libtcmalloc, valgrind to do so, each of them will require a library to be preloaded before your application starts. Doing so, will enable your logging and these logs can be visualized using a profiler.
2. Tracking malloc calls and heap profiling
Here are the steps to use libtcmalloc.
- export LD_PRELOAD=
- Start your application
- export HEAPPROFILE=
- To view stats use pprof –gv
.*
3. Native Memory Tracking(NMT)
It tracks internal memory usage for a HotSpot JVM and can be enabled by:-XX:NativeMemoryTracking=summary
NMT data can be accessed using jcmd utility, you can also view the diff using baseline and diff command.
jcmd
jcmd
jcmd
Limitation with the current version:
NMT does not track third party native code memory allocations and JDK class libraries.
It can result in 5% to 10% JVM performance drop, not advised for production environment.
4. Tracking process memory
To check the heap and non heap memory consumed by JVM use jconsole and use the top –p/ ps v command to track rss .
ps -o rss
Check the memory map for a process and find the diff of two pmap to see the increase in the memory block.
pmap -x
0000000100102000 56K rwx-- [ heap ]
0000000100110000 2624K rwx-- [ heap ] <--- native heap
00000001FB000000 24576K rw--- [ anon ] <--- Java Heap starts here
0000000200000000 1396736K rw--- [ anon ]
0000000600000000 700416K rw--- [ anon ]
diff pmap.1 pmap.9
Use gdb to debug further
gdp -pid [pid]dump memory mem.bin 0x0000000100110000 0x0000000100110000+2624000
5. Java Heap dump analysis (ClassLoader Leaks)
Acquired multiple heap dumps from dev and production environments, used MAT to find class-loader leaks which might cause increase in the metaspace.
Heap dump: jmap -dump:format=b,file=heap_dump.hprof
Thread dump: jstack
Use MAT to check old class-loader instances which did not get garbage collected due to strong references/ link to the gc roots.
Summary
Summarizing, the above steps to debug native memory leaks:
- Monitor the overall memory usage of the process using native tools such as ps, top and pmap.
- Determine whether the memory growth is in the non-Heap regions of the JVM, or in the areas outside the JVM using the JConsole or JMC.
- Force garbage collections and confirm that there is a native memory leak.
- For identifying native leaks from inside the JVM, use NMT.
- For identifying native leaks stemming from outside of the JVM, we can use jemalloc, valgrind, dbx, purify, UMDH, pmap and core files
