Eureka集群的那些坑

今天遇到一个Eureka集群的一个坑。

问题现场类似是这样的：两台Eureka组成的服务注册中心集群，两台服务提供方server1、server2，两个服务调用方client1、client2。

正常的情况下：

client1和client2通过服务中心获取的服务提供方的注册的元数据信息，然后本地缓存调用server2、server1的服务API是没问题的。调用server的api服务时采用的FeignClient通过eureka注册中心获取服务信息调用。

以下简写成client服务、server服务、eureka服务：

但今天遇到的情况是client调用server时，返回结果不稳定，以下是排查过程。

1、习惯性的先查看client方日志，发现接口调用有一定的概率失败，但是并没有抛异常，而仅仅是response返回null。

2、查看server方日志，并没有发现异常，也没有发现response为null的情况，并且server方的本机上访问接口都是响应正常。

3、通过分析两方日志，猜想client端的请求可能没有打到server方。

4、分析整个调用流程，只差eureka服务这一个环节没有排查了，然后查看eureka服务上的server方注册信息是否正常。

5、发现两台eureka服务，有一台tomcat假死了，至此猜想是假死的eureka服务假死所致。

6、验证猜想，重启假死的eureka的tomcat服务，问题验证解决。

那么问题来了，

1、为什么一台eureka假死了，没有抛异常，只在response中返回个null?

2、eureka集群的一台实例假死，为什么会导致服务间调用不稳定？

3、eureka服务跑了大半年，怎么突然一台实例假死了？

接下来，我们来探讨这三个问题。

1、为什么一台eureka假死了，没有抛异常，只在response中返回个null?

这个问题经查，eureka假死，导致调用方调用服务方时，触发了hystrix熔断器,项目实现的熔断降级接口直接在熔断自动降级时返回null，这就回答了第一个问题。

2、eureka集群的一台实例假死，为什么会导致服务间调用不稳定？

这个得翻看一下eureka的机制源码了，猜想feign端在本地缓存90s后去刷新本地服务列表信息时，卡在调用eureka服务上了（此时恰好通过假死的eureka服务刷新），从而服务不可用。

3、eureka服务跑了大半年，怎么突然一台实例假死了？

生产环境有多台虚拟机eureka服务器，配置都一样，其它的服务器没有出过问题，只有一台eureka服务会有类似周期性假死的问题，由于笔者生产环境权限所限，只能猜想可能是宿主机的原因，估计换台机器可能就没问题了，待验证。

还有几个等思考的问题，之后待续。。。。。。。。。。。。。

1、服务提供方配置了多个eureka 地址，是服务注册时，是调用注册到哪个上？ ---> 猜想优化使用第一个，那第二个是何时用到的（除了第一个宕机的情况）

2、feign接口通过eureka调用服务时，是不需要自己项目也注册到eureka上的，怎么处理？

3、服务调用方配置了多个eureka 地址，是获取服务列表调用时，是调用注册到哪个上？ --> 猜想优化使用第一个，那第二个是何时用到的（除了第一个宕机的情况）

另外，昨天查eureka假死日志时，日志打的异常堆栈如下：

2019-06-13 18:07:58.907 INFO 23945 --- [a-EvictionTimer] c.n.e.registry.AbstractInstanceRegistry : Running the evict task with compensationTime 0ms
2019-06-13 18:08:04.949 ERROR 23945 --- [nio-8001-exec-5] o.s.boot.web.support.ErrorPageFilter     : Forwarding to error page from request [/] due to exception [Filter execution threw an exception]

javax.servlet.ServletException: Filter execution threw an exception
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:255) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719) ~[catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:465) ~[catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390) ~[catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317) ~[catalina.jar:8.0.30]
   at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:168) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1257) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1037) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:980) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) ~[servlet-api.jar:na]
   at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[servlet-api.jar:na]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-websocket.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) ~[spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) ~[spring-boot-actuator-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) ~[spring-boot-actuator-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
   at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [catalina.jar:8.0.30]
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.30]
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.30]
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.30]
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.30]
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.30]
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521) [catalina.jar:8.0.30]
   at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096) [tomcat-coyote.jar:8.0.30]
   at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674) [tomcat-coyote.jar:8.0.30]
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-coyote.jar:8.0.30]
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-coyote.jar:8.0.30]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_20]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_20]
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.30]
   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
Caused by: java.lang.NoClassDefFoundError: com/thoughtworks/xstream/io/xml/PrettyPrintWriter
   at com.thoughtworks.xstream.io.xml.DomDriver.createWriter(DomDriver.java:122) ~[xstream-1.4.9.jar:1.4.9]
   at com.thoughtworks.xstream.io.xml.DomDriver.createWriter(DomDriver.java:127) ~[xstream-1.4.9.jar:1.4.9]
   at com.thoughtworks.xstream.XStream.toXML(XStream.java:1042) ~[xstream-1.4.9.jar:1.4.9]
   at com.netflix.discovery.converters.wrappers.CodecWrappers$XStreamXml.encode(CodecWrappers.java:377) ~[eureka-client-1.4.12.jar:1.4.12]
   at com.netflix.discovery.provider.DiscoveryJerseyProvider.writeTo(DiscoveryJerseyProvider.java:135) ~[eureka-client-1.4.12.jar:1.4.12]
   at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302) ~[jersey-server-1.19.1.jar:1.19.1]
   at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) ~[jersey-server-1.19.1.jar:1.19.1]
   at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.1.jar:1.19.1]
   at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.1.jar:1.19.1]
   at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.1.jar:1.19.1]
   at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.1.jar:1.19.1]
   at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:927) ~[jersey-servlet-1.19.1.jar:1.19.1]
   at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) ~[jersey-servlet-1.19.1.jar:1.19.1]
   at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) ~[jersey-servlet-1.19.1.jar:1.19.1]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
   ... 93 common frames omitted

2019-06-13 18:08:58.907 INFO 23945 --- [a-EvictionTimer] c.n.e.registry.AbstractInstanceRegistry : Running the evict task with compensationTime 0ms
2019-06-13 18:09:42.160 INFO 23945 --- [hresholdUpdater] c.n.e.r.PeerAwareInstanceRegistryImpl    : Current renewal threshold is : 102
2019-06-13 18:09:58.907 INFO 23945 --- [a-EvictionTimer] c.n.e.registry.AbstractInstanceRegistry : Running the evict task with compensationTime 0ms
2019-06-13 18:10:19.760 INFO 23945 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration

这个其实就是找不到报错了找不到相应的url所致，可以设置com.ibm.ws.webcontainer.invokeFlushAfterService为false，关闭此日志。

网上也有说可以增加如下依赖解决关闭日志，没试过，此处只记录一下。

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-thymeleaf</artifactId>
   <version>${spring.boot.version}</version>
</dependency>

<!--HTML扫描器和标签补偿器-->
<dependency>
   <groupId>net.sourceforge.nekohtml</groupId>
   <artifactId>nekohtml</artifactId>
   <version>1.9.22</version>
</dependency>

posted @ 2019-06-13 20:51 xuzhujack 阅读(23592) 评论(2) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

xuzhujack

Walk the road you want to walk and do what you want to do , keep moving ahead and ... ...

Eureka集群的那些坑

公告