从容器里dump java堆实验探索(原创)

目标：从docker容器里dump java堆

 

模拟程序

 

 

占用空间500M,

设置启动JVM参数

 

 

 

docker启动命令

 

 

（PS：经过测试，至少要650M才能启动容器）

 

 

方式1: 通过docker exec

先通过 docker exec $containerid ps x 获取进程号

执行 docker exec

 

 

此命令实际就是进入docker容器里执行/jdk/bin/jmap dump ，导出的文件也是存放在容器里

经过多次实验，基本上每次都触发容器killed

方式2,: 通过nsenter

参考文章 https://github.com/jpetazzo/nsenter

 

这里有句重点是”evades resource limitations” ，躲避资源限制

我们使用nsenter试验一下

使用docker-enter  (docker-enter脚本 调用了nsenter)

 

 

多次测试，都没有发生docker killed现象

 

 

结论: 通过nsenter 方式去 dump java 堆，能比较高成功率， 但仅限于此实验场景，不代表其他线上场景一定能成功

 

PS: jdk10 新特性里虽然增加了更好的docker支持，但没有提及过jmap 有更好的dump方式

https://bugs.openjdk.java.net/browse/JDK-8146115

 

To correct these shortcomings and make this support more robust, here's a list of the current cgroup subsystems that we be examined in order to update the internal VM and core library configuration. 

Number of CPUs 
----------------------- 
Use a combination of number_of_cpus() and cpu_sets() in order to determine how many processors are available to the process and adjust the JVMs os::active_processor_count appropriately. The number_of_cpus() will be calculated based on the cpu_quota() and cpu_period() using this formula: number_of_cpus() = cpu_quota() / cpu_period(). If cpu_shares has been setup for the container, the number_of_cpus() will be calculated based on cpu_shares()/1024. 1024 is the default and standard unit for calculating relative cpu usage in cloud based container management software. 

Also add a new VM flag (-XX:ActiveProcessorCount=xx) that allows the number of CPUs to be overridden. This flag will be honored even if UseContainerSupport is not enabled. 

Total available memory 
------------------------------- 
Use the memory_limit() value from the cgroup file system to initialize the os::physical_memory() value in the VM. This value will propagate to all other parts of the Java runtime. 

Memory usage 
-------------------- 
Use memory_usage_in_bytes() for providing os::available_memory() by subtracting the usage from the total available memory allocated to the container. 

As as troubleshooting aid, we will dump any available container statistics to the hotspot error log and add container specific information to the JVM logging system. Unified Logging will be added to help to diagnose issue related to this support. Use -Xlog:os+container=trace for maximum logging of container information. 

A new option -XX:-UseContainerSupport will be added to allow the container support to be disabled. The default for this flag will be true. Container support will be enabled by default. 

 

 

PS: jmap –F 那点事

When run without -F these tools use Dynamic Attach Mechanism. This works as follows.

1. Before connecting to Java process 1234, jmap creates a file .attach_pid1234 at the working directory of the target process or at /tmp.
2. Then jmap sends SIGQUIT to the target process. When JVM catches the signal and finds .attach_pid1234, it starts AttachListener thread.
3. AttachListener thread creates UNIX domain socket /tmp/.java_pid1234 to listen to commands from external tools.
4. For security reasons when a connection (from jmap) is accepted, JVM verifies that credentials of the socket peer are equal to euid and egid of JVM process. That's why jmap will not work if run by different user (even by root).
5. jmap connects to the socket, and sends dumpheap command.
6. This command is read and executed by AttachListener thread of the JVM. All output is sent back to the socket. Since the heap dump is made in-process directly by JVM, the operation is really fast. However, JVM can do this only at safepoints. If a safepoint cannot be reached (e.g. the process is hung, not responding, or a long GC is in progress), jmap will timeout and fail.

Let's summarize the benefits and the drawbacks of Dynamic Attach.

Pros.

• Heap dump and other operations are run collaboratively by JVM at the maximum speed.
• You can use any version of jmap or jstack to connect to any other version of JVM.

Cons.

• The tool should be run by the same user (euid/egid) as the target JVM.
• Can be used only on live and healthy JVM.
• Will not work if the target JVM is started with -XX:+DisableAttachMechanism.

---

jmap -F / jstack -F

When run with -F the tools switch to special mode that features HotSpot Serviceability Agent. In this mode the target process is frozen; the tools read its memory via OS debugging facilities, namely, ptrace on Linux.

1. jmap -F invokes PTRACE_ATTACH on the target JVM. The target process is unconditionally suspended in response to SIGSTOP signal.
2. The tool reads JVM memory using PTRACE_PEEKDATA. ptrace can read only one word at a time, so too many calls required to read the large heap of the target process. This is very and very slow.
3. The tool reconstructs JVM internal structures based on the knowledge of the particular JVM version. Since different versions of JVM have different memory layout, -F mode works only if jmap comes from the same JDK as the target Java process.
4. The tool creates heap dump itself and then resumes the target process.

Pros.

• No cooperation from target JVM is required. Can be used even on a hung process.
• ptrace works whenever OS-level privileges are enough. E.g. root can dump processes of all other users.

Cons.

• Very slow for large heaps.
• The tool and the target process should be from the same version of JDK.

The safepoint is not guaranteed when the tool attaches in forced mode. Though jmap tries to handle all special cases, sometimes it may happen that target JVM is not in a consistent state.

 

 

方式3: 直接在宿主机执行jmap

首先要先获取到容器里映射到宿主极的进程号

 

 

21025就是映射到宿主机的进程id

 

调用jmap –F 21025 ,但是失败了

 

 

 

没有任何头绪之际,突然发现

 

 

 

exe -> /jdk/bin/java 是红色, 这是容器里的java路径, 但宿主机没有这个路径, 会不会是这里有影响, 赶紧在宿主机也创建这个路径

 

 

 

再次执行jmap, 成功导出