记一次Java Core Dump分析过程

背景提要

---

很久没有亲自动手部署代码了，命令行的亲切感越来越低。放飞了键盘，习惯了鼠标操作的windows环境。冷不丁实操部署也是不错的。

常常在部署时，运维同学对于[hs_err_pid]文件视而不见。殊不知这是Java 虚拟机崩溃日志。

这次是如何分析问题的？

---

一、首先查看日志头文件

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f90e71183a8, pid=6167, tid=0x00007f8fab3e6700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x1503a8]  __memmove_ssse3_back+0x138
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

日志头文件包含概要信息，简述了导致crash的原因。

The crash happened outside the Java Virtual Machine in native code.

能明显看到本次崩溃是因为JNI。

二、接下来查看崩溃时线程信息

---------------  T H R E A D  ---------------

Current thread (0x00007f90cc03f800):  JavaThread "restartedMain" [_thread_in_native, id=23577, stack(0x00007f8fab2e6000,0x00007f8fab3e7000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f90e7be5000

Registers:
RAX=0x00007f8fcc04e4f2, RBX=0x00007f90e7be34e2, RCX=0x00007f8fcc04e4f0, RDX=0x00000000000013d5
RSP=0x00007f8fab3e2c68, RBP=0x00007f8fab3e2cc0, RSI=0x00007f90e7be4ff0, RDI=0x00007f8fcc050000
R8 =0x00007f8fcc04e4f2, R9 =0x0000000000002f84, R10=0x0000000000000070, R11=0x00007f90e714c6d0
R12=0x00007f90e7bdd085, R13=0x00007f8fcc014390, R14=0x0000000000002f63, R15=0x00007f90e021f5e0
RIP=0x00007f90e71183a8, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f8fab3e2c68)
0x00007f8fab3e2c68:   00007f90e5065c09 00007f90d1046960
0x00007f8fab3e2c78:   6863639000000000 000000000000642f
0x00007f8fab3e2c88:   00000000014b502f 00007f8fcc014390
0x00007f8fab3e2c98:   00007f90e021f7d0 00007f900c01da90
0x00007f8fab3e2ca8:   00007f90e021f5e0 000000004d79b210
0x00007f8fab3e2cb8:   000000000000000d 00007f8fab3e2d20
0x00007f8fab3e2cc8:   00007f90e506618f 00007f90cc03f800
0x00007f8fab3e2cd8:   01007f902466f0c0 00007f8fab3e2d40
0x00007f8fab3e2ce8:   00007f8fab3e2d4e 00007f8fab3e2d4c
0x00007f8fab3e2cf8:   00007f8fab3e2d40 000000000000000d
0x00007f8fab3e2d08:   00007f90cc03f9f8 00007f8fab3e3198
0x00007f8fab3e2d18:   0000000000000001 00007f8fab3e3180
0x00007f8fab3e2d28:   00007f90e5057ec0 00007f8fab3e2d70
0x00007f8fab3e2d38:   00007f90e021f5e0 632f63642f6d6f63
0x00007f8fab3e2d48:   0000002f6e696168 00007f90bb0423d8
0x00007f8fab3e2d58:   00007f90cc03f800 00007f8fab3e2db0
0x00007f8fab3e2d68:   00007f8fab3e2e30 00007f90bb0423d8
0x00007f8fab3e2d78:   00007f90cc03f800 00007f8fab3e2db0
0x00007f8fab3e2d88:   00007f90cc02e850 000000076f6d3910
0x00007f8fab3e2d98:   00007f90bb0423d8 00007f90cc03d800
0x00007f8fab3e2da8:   00007f90cc03f800 00007f8fab3e2f40
0x00007f8fab3e2db8:   00007f90e66a739f 00007f90d2ad3d50
0x00007f8fab3e2dc8:   0000000000000000 00007f8fab3e2e20
0x00007f8fab3e2dd8:   00007f8fab3e2ea0 00007f8fab3e2f70
0x00007f8fab3e2de8:   00007f8fab3e2ff0 00007f90cc03f9f8
0x00007f8fab3e2df8:   00007f90e66c64d0 00007f9004028a70
0x00007f8fab3e2e08:   00007f9004028a80 00007f9004028e58
0x00007f8fab3e2e18:   00000000000003d8 00007f8fab3e35a0
0x00007f8fab3e2e28:   000000076f6d3910 00000006c7c1da60
0x00007f8fab3e2e38:   00007f90d2ad4078 00007f90d2ad3d50
0x00007f8fab3e2e48:   0000000000000000 00007f8fab3e3740
0x00007f8fab3e2e58:   00007f8fab3e3680 00007f90cc03f800

从线程名已经缺定了JavaThread "restartedMain"。调用重启方法线程，触发了崩溃。

三、再来查看崩溃时线程栈信息

Stack: [0x00007f8fab2e6000,0x00007f8fab3e7000],  sp=0x00007f8fab3e2c68,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x1503a8]  __memmove_ssse3_back+0x138
C  [libzip.so+0x1218f]  ZIP_GetEntry2+0xff
C  [libzip.so+0x3ec0]  Java_java_util_zip_ZipFile_getEntry+0xf0
J 19  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f90d110140e [0x00007f90d1101340+0xce]
J 6501 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (101 bytes) @ 0x00007f90d1d10708 [0x00007f90d1d10380+0x388]
J 11361 C2 java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (22 bytes) @ 0x00007f90d13b7018 [0x00007f90d13b6fe0+0x38]
J 11334 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f90d11ba604 [0x00007f90d11ba580+0x84]
J 5624 C2 sun.misc.URLClassPath$1.next()Z (63 bytes) @ 0x00007f90d2299ae0 [0x00007f90d22998c0+0x220]
J 29485 C2 java.net.URLClassLoader$3$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f90d34e1090 [0x00007f90d34e1040+0x50]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x693e96]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x72d16c]  JVM_DoPrivileged+0x27c
J 1053  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d1460095 [0x00007f90d145ffc0+0xd5]
J 25745 C2 java.net.URLClassLoader$3.hasMoreElements()Z (5 bytes) @ 0x00007f90d54a81a4 [0x00007f90d54a8100+0xa4]
J 29484 C2 sun.misc.CompoundEnumeration.hasMoreElements()Z (5 bytes) @ 0x00007f90d4b42620 [0x00007f90d4b425a0+0x80]
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.doFindAllClassPathResources(Ljava/lang/String;)Ljava/util/Set;+35
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+19
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+52
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+17
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+38
j  org.springframework.boot.devtools.restart.ClassLoaderFilesResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+13
j  org.springframework.context.support.GenericApplicationContext.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+18
j  org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(Ljava/lang/String;)Ljava/util/Set;+49
j  org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan([Ljava/lang/String;)Ljava/util/Set;+39
j  org.springframework.context.annotation.ComponentScanAnnotationParser.parse(Lorg/springframework/core/annotation/AnnotationAttributes;Ljava/lang/String;)Ljava/util/Set;+517
J 25607 C1 org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass;)Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass; (515 bytes) @ 0x00007f90d17aa93c [0x00007f90d17aa220+0x71c]
J 3183 C1 org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;)V (143 bytes) @ 0x00007f90d1c875ac [0x00007f90d1c870e0+0x4cc]
j  org.springframework.context.annotation.ConfigurationClassParser.parse(Lorg/springframework/core/type/AnnotationMetadata;Ljava/lang/String;)V+10
j  org.springframework.context.annotation.ConfigurationClassParser.parse(Ljava/util/Set;)V+66
j  org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+291
j  org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+107
j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(Ljava/util/Collection;Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+28
j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;Ljava/util/List;)V+229
j  org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;)V+5
j  org.springframework.context.support.AbstractApplicationContext.refresh()V+28
j  org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh()V+1
j  org.springframework.boot.SpringApplication.refresh(Lorg/springframework/context/ApplicationContext;)V+11
j  org.springframework.boot.SpringApplication.refreshContext(Lorg/springframework/context/ConfigurableApplicationContext;)V+2
j  org.springframework.boot.SpringApplication.run([Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+91
j  org.springframework.boot.SpringApplication.run([Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9
j  org.springframework.boot.SpringApplication.run(Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9
j  com.dc.chain.Application.main([Ljava/lang/String;)V+31
v  ~StubRoutines::call_stub
V  [libjvm.so+0x693e96]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x9a0447]  Reflection::invoke(instanceKlassHandle, methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*)+0x5d7
V  [libjvm.so+0x9a3b57]  Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0x147
V  [libjvm.so+0x72754b]  JVM_InvokeMethod+0x26b
J 1920  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d17c586d [0x00007f90d17c5780+0xed]
J 12554 C2 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007f90d27308f4 [0x00007f90d2730880+0x74]
J 31113 C2 org.springsource.loaded.ri.OriginalClassInvoker.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (17 bytes) @ 0x00007f90d3671c98 [0x00007f90d3671be0+0xb8]
J 25653 C2 org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (2214 bytes) @ 0x00007f90d23f1a5c [0x00007f90d23f1380+0x6dc]
j  org.springframework.boot.devtools.restart.RestartLauncher.run()V+41
v  ~StubRoutines::call_stub
V  [libjvm.so+0x693e96]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x6943a1]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x694847]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x72fbd0]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0xa79753]  JavaThread::thread_main_inner()+0x103
V  [libjvm.so+0xa7989c]  JavaThread::run()+0x11c
V  [libjvm.so+0x92a338]  java_start(Thread*)+0x108
C  [libpthread.so.0+0x7e25]  start_thread+0xc5

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 19  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f90d1101398 [0x00007f90d1101340+0x58]
J 6501 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (101 bytes) @ 0x00007f90d1d10708 [0x00007f90d1d10380+0x388]
J 11361 C2 java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (22 bytes) @ 0x00007f90d13b7018 [0x00007f90d13b6fe0+0x38]
J 11334 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f90d11ba604 [0x00007f90d11ba580+0x84]
J 5624 C2 sun.misc.URLClassPath$1.next()Z (63 bytes) @ 0x00007f90d2299ae0 [0x00007f90d22998c0+0x220]
J 29485 C2 java.net.URLClassLoader$3$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f90d34e1090 [0x00007f90d34e1040+0x50]
v  ~StubRoutines::call_stub
J 1053  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d1460023 [0x00007f90d145ffc0+0x63]
J 25745 C2 java.net.URLClassLoader$3.hasMoreElements()Z (5 bytes) @ 0x00007f90d54a81a4 [0x00007f90d54a8100+0xa4]
J 29484 C2 sun.misc.CompoundEnumeration.hasMoreElements()Z (5 bytes) @ 0x00007f90d4b42620 [0x00007f90d4b425a0+0x80]
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.doFindAllClassPathResources(Ljava/lang/String;)Ljava/util/Set;+35
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+19
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+52
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+17
j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+38
j  org.springframework.boot.devtools.restart.ClassLoaderFilesResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+13
j  org.springframework.context.support.GenericApplicationContext.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+18
j  org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(Ljava/lang/String;)Ljava/util/Set;+49
j  org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan([Ljava/lang/String;)Ljava/util/Set;+39
j  org.springframework.context.annotation.ComponentScanAnnotationParser.parse(Lorg/springframework/core/annotation/AnnotationAttributes;Ljava/lang/String;)Ljava/util/Set;+517
J 25607 C1 org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass;)Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass; (515 bytes) @ 0x00007f90d17aa93c [0x00007f90d17aa220+0x71c]
J 3183 C1 org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;)V (143 bytes) @ 0x00007f90d1c875ac [0x00007f90d1c870e0+0x4cc]
j  org.springframework.context.annotation.ConfigurationClassParser.parse(Lorg/springframework/core/type/AnnotationMetadata;Ljava/lang/String;)V+10
j  org.springframework.context.annotation.ConfigurationClassParser.parse(Ljava/util/Set;)V+66
j  org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+291
j  org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+107
j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(Ljava/util/Collection;Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+28
j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;Ljava/util/List;)V+229
j  org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;)V+5
j  org.springframework.context.support.AbstractApplicationContext.refresh()V+28
j  org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh()V+1
j  org.springframework.boot.SpringApplication.refresh(Lorg/springframework/context/ApplicationContext;)V+11
j  org.springframework.boot.SpringApplication.refreshContext(Lorg/springframework/context/ConfigurableApplicationContext;)V+2
j  org.springframework.boot.SpringApplication.run([Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+91
j  org.springframework.boot.SpringApplication.run([Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9
j  org.springframework.boot.SpringApplication.run(Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9
j  com.dc.chain.Application.main([Ljava/lang/String;)V+31
v  ~StubRoutines::call_stub
J 1920  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d17c57f7 [0x00007f90d17c5780+0x77]
J 12554 C2 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007f90d27308f4 [0x00007f90d2730880+0x74]
J 31113 C2 org.springsource.loaded.ri.OriginalClassInvoker.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (17 bytes) @ 0x00007f90d3671c98 [0x00007f90d3671be0+0xb8]
J 25653 C2 org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (2214 bytes) @ 0x00007f90d23f1a5c [0x00007f90d23f1380+0x6dc]
j  org.springframework.boot.devtools.restart.RestartLauncher.run()V+41
v  ~StubRoutines::call_stub

明显看到栈顶调用为： Java_java_util_zip_ZipFile_getEntry+0xf0

四、根据栈顶信息查询

当然是用的某歌搜索引擎，查到如下地址：
(Crash in [libzip.so+0x1218f] ZIP_GetEntry2+0xff)[https://bugs.openjdk.java.net/browse/JDK-8175970]

查询到结果，可以知道在JDK 9中修正了这个Bug。
This is fixed in JDK 9 with JDK-8145260.
Most of the times, the crashes in ZIP_GetEntry occur when the jar file being accessed has been modified/overwritten while the JVM instance was running. The following property can be used to disable the memory mapping of the central directory structure of the Jar files:
-Dsun.zip.disableMemoryMapping=true
Please note that enabling this property would have some performance impact on the application as the JVM needs to read the central directory structure from the Jar files on the disk again and again whenever it reads a Jar file entry. So, it is best to ensure that the jar files are not modified or overwritten while the JVM has an image of them loaded.

For details refer the following :
https://blogs.oracle.com/poonam/entry/crashes_in_zip_getentry