tomcat outMemoy后服务器假死了
最近融360开量后core服务器cpu瞬间飙到200%，而且middle服务器会报outofmemory，开启tomcat日志后middleB看到的是这样的：
[16/Feb/2019:11:26:57 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:27:02 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:26:59 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:27:07 +0800] "POST /rongApi/addInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:27:07 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:27:07 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:27:16 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:27:23 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:11:27:25 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
[16/Feb/2019:12:48:40 +0800] "POST /qianchengApi/doCall HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:40 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:41 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:41 +0800] "POST /rongApi/applyCheck HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:41 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:41 +0800] "POST /qianchengApi/doCall HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:41 +0800] "POST /rongApi/loanCalculate HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:11:04:49 +0800] "POST /rongApi/loanCalculate HTTP/1.0" 500 - [16/Feb/2019:12:48:41 +0800] "POST /rongApi/applyCheck HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:41 +0800] "POST /rongApi/applyCheck HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:42 +0800] "POST /rongApi/getRepaymentPlan HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:42 +0800] "POST /rongApi/applyCheck HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:42 +0800] "POST /rongApi/approvalConfirmation HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:42 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalStateException: Failed to introspect annotations on class org.springframework.web.util.NestedServletException [16/Feb/2019:12:13:02 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 500 - [16/Feb/2019:12:48:42 +0800] "POST /rongApi/getRongReport HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:42 +0800] "POST /rongApi/loanCalculate HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:42 +0800] "POST /rongApi/getApprovalConclusion HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:42 +0800] "POST /rongApi/applyCheck HTTP/1.0" 500 org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space [16/Feb/2019:12:48:49 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 - [16/Feb/2019:12:48:51 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:51 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:53 +0800] "POST /rongApi/baseInfoPush HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:57 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:57 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:58 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:59 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:48:58 +0800] "POST /rongApi/loanCalculate HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:01 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:04 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:04 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:04 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:04 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:04 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:05 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:07 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:49:09 +0800] "POST /rongApi/getRepaymentPlan HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pi 。。。。。。 。。。。。。 [16/Feb/2019:12:50:44 +0800] "POST /qianchengApi/doCall HTTP/1.0" 200 org.apache.catalina.connector.ClientAbo [16/Feb/2019:12:50:44 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbor [16/Feb/2019:12:50:44 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbor [16/Feb/2019:12:50:44 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbor [16/Feb/2019:12:50:44 +0800][16/Feb/2019:13:37:59 +0800] "POST /gnxjdApi/helipayQuickPayNotify HTTP/1.0" 200 [16/Feb/2019:13:38:02 +0800] "POST /qianchengApi/doCall HTTP/1.0" 200 - [16/Feb/2019:13:38:05 +0800] "OPTIONS /qianchengApi/authPage HTTP/1.0" 200 - [16/Feb/2019:13:38:05 +0800] "POST /qianchengApi/authPage HTTP/1.0" 200 - [16/Feb/2019:12:50:44 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe [16/Feb/2019:12:50:44 +0800][16/Feb/2019:13:37:59 +0800] "POST /gnxjdApi/helipayQuickPayNotify HTTP/1.0" 200 - [16/Feb/2019:13:38:02 +0800] "POST /qianchengApi/doCall HTTP/1.0" 200 - [16/Feb/2019:13:38:05 +0800] "OPTIONS /qianchengApi/authPage HTTP/1.0" 200 - [16/Feb/2019:13:38:05 +0800] "POST /qianchengApi/authPage HTTP/1.0" 200 - [16/Feb/2019:13:38:07 +0800] "POST /rongApi/getRepaymentPlan HTTP/1.0" 200 - [16/Feb/2019:13:38:08 +0800] "POST /rongApi/getOrderStatus HTTP/1.0" 200 - [16/Feb/2019:13:38:09 +0800] "POST /rongApi/getRepaymentPlan HTTP/1.0" 200 - [16/Feb/2019:13:38:10 +0800] "POST /rongApi/getRepaymentPlan HTTP/1.0" 200 - [16/Feb/2019:13:38:10 +0800] "POST /qianchengApi/doCall HTTP/1.0" 200 - [16/Feb/2019:13:38:11 +0800] "POST /rongApi/getRepaymentPlan HTTP/1.0" 200 - [16/Feb/2019:13:38:11 +0800] "POST /rongApi/getContracts HTTP/1.0" 200 - [16/Feb/2019:13:38:11 +0800] "POST /rongApi/bankCardList HTTP/1.0" 200 - [16/Feb/2019:13:38:13 +0800] "POST /rongApi/applyCheck HTTP/1.0" 200 -
对应middleB的应用日志是：
019-02-16 12:48:49,962 com.ps.gnxjd.middle.controller.RongApiController [http-nio-10031-exec-2983] (RongApiController.java:79) INFO --> applyCheck-应答:{"code":400,"msg":"用户不可以申请","reason":"C001"}
2019-02-16 12:48:51,098 com.ps.gnxjd.third.rong.RongApiService [userinfoRong360Task-130] (RongApiService.java:488) ERROR --> Java heap space
com.alibaba.fastjson.JSONException: Java heap space
at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:614)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:327)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:295)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:209)
at com.ps.gnxjd.utils.JsonUtil.getObject(JsonUtil.java:35)
at com.ps.gnxjd.third.rong.RongApiService.getRongParameter(RongApiService.java:470)
at com.ps.gnxjd.middle.controller.RongUserinfoAsyncService.dealInfoPush(RongUserinfoAsyncService.java:57)
at com.ps.gnxjd.middle.controller.RongUserinfoAsyncService$$FastClassBySpringCGLIB$$c98f6f62.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2019-02-16 12:48:51,098 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-130] (RongUserinfoAsyncService.java:72) ERROR --> 应答:{"code":-140,"msg":"非法请求"}
2019-02-16 12:48:51,098 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-130] (RongUserinfoAsyncService.java:73) ERROR --> 非法请求
com.ps.gnxjd.third.rong.RongException: 非法请求
at com.ps.gnxjd.third.rong.RongApiService.getRongParameter(RongApiService.java:489)
at com.ps.gnxjd.middle.controller.RongUserinfoAsyncService.dealInfoPush(RongUserinfoAsyncService.java:57)
at com.ps.gnxjd.middle.controller.RongUserinfoAsyncService$$FastClassBySpringCGLIB$$c98f6f62.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
555 2019-02-16 12:50:20,723 com.ps.gnxjd.middle.controller.RongApiController [http-nio-10031-exec-2590] (RongApiController.java:85) ERROR --> applyCheck-应答:{"code":-99,"msg":"系统繁忙"}
556 2019-02-16 12:51:02,973 com.ps.gnxjd.middle.controller.RongApiController [http-nio-10031-exec-2590] (RongApiController.java:87) ERROR -->
557 java.util.concurrent.TimeoutException
558 at java.util.concurrent.FutureTask.get(FutureTask.java:205)
559 at com.ps.gnxjd.middle.controller.RongApiController.applyCheck(RongApiController.java:77)
560 at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
561 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
562 at java.lang.reflect.Method.invoke(Method.java:498)
563 at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
564 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
565 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
566 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
567 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
568 at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
569 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
570 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
571 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
572 at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
573 at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
574 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
575 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
576 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
577 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
578 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
579 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
580 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
581 at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
582 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
583 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
584 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
585 at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
586 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
587 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
588 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
809 java.util.concurrent.TimeoutException 810 at java.util.concurrent.FutureTask.get(FutureTask.java:205) 811 at com.ps.gnxjd.middle.controller.RongApiController.applyCheck(RongApiController.java:77) 812 at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source) 813 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 814 at java.lang.reflect.Method.invoke(Method.java:498) 815 at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) 816 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) 817 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) 818 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) 819 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) 820 at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) 821 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) 822 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) 823 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) 824 at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) 825 at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) 826 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) 827 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) 828 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) 829 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 830 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) 831 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 832 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 833 at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) 834 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 835 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 836 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 837 at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) 838 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 839 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 840 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 841 at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:108) 842 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 843 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 844 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 845 at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) 846 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 847 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 848 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 849 at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) 850 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 851 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 852 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 853 at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) 854 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
。。。。。。。。。。
8707 2019-02-16 12:52:06,747 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-121] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8708 2019-02-16 12:52:06,839 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-138] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8709 2019-02-16 12:52:06,880 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-139] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8710 2019-02-16 12:52:06,930 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-125] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8711 2019-02-16 12:52:06,945 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-129] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8712 2019-02-16 12:52:06,983 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-141] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8713 2019-02-16 12:52:07,007 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-131] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8714 2019-02-16 12:52:07,037 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-130] (RongUserinfoAsyncService.java:84) INFO --> baseInfoPush-应答:{"code":200,"msg":"成功"} 8715 2019-02-16 12:52:07,128 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-123] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8716 2019-02-16 12:52:07,175 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-136] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8717 2019-02-16 12:52:07,273 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-134] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8718 2019-02-16 12:52:07,300 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-116] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8719 2019-02-16 12:52:07,336 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-128] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8720 2019-02-16 12:52:07,395 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-135] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8721 2019-02-16 12:52:08,294 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-122] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8722 2019-02-16 12:52:09,039 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-120] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8723 2019-02-16 12:52:09,044 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-137] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8724 2019-02-16 12:52:09,069 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-126] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8725 2019-02-16 12:52:10,364 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-140] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8726 2019-02-16 12:52:10,689 com.ps.gnxjd.middle.controller.RongUserinfoAsyncService [userinfoRong360Task-38] (RongUserinfoAsyncService.java:84) INFO --> addInfoPush-应答:{"code":200,"msg":"成功"} 8727 2019-02-16 13:00:37,196 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration 8728 2019-02-16 13:05:37,196 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration 8729 2019-02-16 13:06:40,889 net.sf.ehcache.store.disk.DiskStorageFactory [user.data] (DiskStorageFactory.java:488) ERROR --> Disk Write of d943460ca7e6446d8b67a93e5d2d6d6e failed: 8730 java.lang.OutOfMemoryError: Java heap space 8731 2019-02-16 13:06:40,889 net.sf.ehcache.store.disk.DiskStorageFactory [user.data] (DiskStorageFactory.java:488) ERROR --> Disk Write of d943460ca7e6446d8b67a93e5d2d6d6e failed: 8732 java.lang.OutOfMemoryError: Java heap space 8733 2019-02-16 13:10:37,196 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration 8734 2019-02-16 13:14:28,942 net.sf.ehcache.store.disk.DiskStorageFactory [user.data] (DiskStorageFactory.java:488) ERROR --> Disk Write of 4f069981aac540ca8770272ee6a5d9bd failed: 8735 java.lang.OutOfMemoryError: Java heap space 8736 2019-02-16 13:14:28,942 net.sf.ehcache.store.disk.DiskStorageFactory [user.data] (DiskStorageFactory.java:488) ERROR --> Disk Write of 4f069981aac540ca8770272ee6a5d9bd failed: 8737 java.lang.OutOfMemoryError: Java heap space 8738 2019-02-16 13:15:37,197 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration 8739 2019-02-16 13:20:37,197 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration
8739 2019-02-16 13:20:37,197 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration
8740 2019-02-16 13:25:37,198 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration
8741 2019-02-16 13:30:37,198 com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver [AsyncResolver-bootstrap-executor-0] (ConfigClusterResolver.java:43) INFO --> Resolving eureka endpoints via configuration
8742 2019-02-16 13:33:14,061 com.netflix.loadbalancer.PollingServerListUpdater [Thread-15] (PollingServerListUpdater.java:53) INFO --> Shutting down the Executor Pool for PollingServerListUpdater
8743 2019-02-16 13:33:14,061 org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext [Thread-12] (AbstractApplicationContext.java:987) INFO --> Closing org.springframework.boot.context.embedded.AnnotationCon figEmbeddedWebApplicationContext@3d1848cc: startup date [Thu Feb 14 16:33:19 CST 2019]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@4facf68f
8744 2019-02-16 13:33:14,063 org.springframework.cloud.netflix.eureka.EurekaDiscoveryClientConfiguration [Thread-12] (EurekaDiscoveryClientConfiguration.java:128) INFO --> Unregistering application service-middle with eureka with status DO WN
8745 2019-02-16 13:33:14,064 com.netflix.discovery.DiscoveryClient [Thread-12] (DiscoveryClient.java:1294) WARN --> Saw local status change event StatusChangeEvent [timestamp=1550295194064, current=DOWN, previous=UP]
8746 2019-02-16 13:33:14,064 com.netflix.discovery.DiscoveryClient [DiscoveryClient-InstanceInfoReplicator-0] (DiscoveryClient.java:812) INFO --> DiscoveryClient_SERVICE-MIDDLE/izuf68tmr3yb0no2tcbddnz:service-middle:10031: registering serv ice...
8747 2019-02-16 13:33:14,064 com.netflix.discovery.DiscoveryClient [Thread-12] (DiscoveryClient.java:869) INFO --> Shutting down DiscoveryClient ...
8748 2019-02-16 13:33:14,065 com.netflix.discovery.DiscoveryClient [Thread-12] (DiscoveryClient.java:901) INFO --> Unregistering ...
8749 2019-02-16 13:33:14,074 com.netflix.discovery.DiscoveryClient [DiscoveryClient-InstanceInfoReplicator-0] (DiscoveryClient.java:821) INFO --> DiscoveryClient_SERVICE-MIDDLE/izuf68tmr3yb0no2tcbddnz:service-middle:10031 - registration st atus: 204
854,5-12 8%
从tomcat日志中可以看到，11:26:57 报了Java heap space，12:50:44到13:38:02没有处理任何请求，从应用日志中看到，在12:50:20分futureTask发生了timeout异常，12:48:51解析json发生 Java heap space， 13:06:40报了outofmenory。
总结：middleB 发生了outofmenory，原因可能是core层处理请求太慢，导致middle层积累了很多请求数据，每个基本信息，补充信息比较大，解析数据时报内存溢出，溢出后服务器服务就挂了。
用MAT分析堆栈。
喜欢艺术的码农
