一次mysql驱动版本不一致导致的问题

现象

我们发版都是多组服务器每次发一组,验一组,然后nginx停掉流量保证用户不会请求到这组机器,同时dubbo内部做版本隔离

1.某天发版出现大量异常奇怪的除了新版本报错,老版本也报这个错

当时很担心无法做回退,尝试停掉新的发老的就不再报错,因为考虑到全发新的启动时间太长,影响用户使用选择了放弃发版后续测试环境模拟复现排查,庆幸测试环境按照同样方式复现了

[DubboServerHandler-10.42.5.55:20930-thread-5] 709810 ERROR 2022-05-30 10:49:03,553 com.ewei.support.spring.hibernate.HibernateTemplate:49 <0><T1531105378825478144> doExecute error, sql=SELECT provider0_.id AS id1_88_0_, provider0_.alone_domain AS alone_do2_88_0_, provider0_.contact_name AS contact_3_88_0_, provider0_.contact_phone AS contact_4_88_0_, provider0_.created_at AS created_5_88_0_
    , provider0_.employees AS employee6_88_0_, provider0_.independent AS independ7_88_0_, provider0_.is_alone_domain_checked AS is_alone8_88_0_, provider0_.is_info_completed AS is_info_9_88_0_, provider0_.is_open_helpcenter AS is_open10_88_0_
    , provider0_.is_open_resources AS is_open11_88_0_, provider0_.is_open_sms_notify AS is_open12_88_0_, provider0_.name AS name13_88_0_, provider0_.phone AS phone14_88_0_, provider0_.record_number AS record_15_88_0_
    , provider0_.sub_domain AS sub_dom16_88_0_, provider0_.update_at AS update_17_88_0_, provider0_.valid AS valid18_88_0_, provider0_.wx_cropid AS wx_crop19_88_0_
FROM provider provider0_
WHERE provider0_.id = 7893
org.springframework.orm.hibernate4.HibernateObjectRetrievalFailureException: No row with the given identifier exists: [com.ewei.account.api.entity.Provider#7893]; nested exception is org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.ewei.account.api.entity.Provider#7893]
    at org.springframework.orm.hibernate4.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:194)
    at org.springframework.orm.hibernate4.HibernateTemplate.doExecute(HibernateTemplate.java:343)
    at com.ewei.support.spring.hibernate.HibernateTemplate.doExecute(HibernateTemplate.java:32)
    at org.springframework.orm.hibernate4.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:308)
    at org.springframework.orm.hibernate4.HibernateTemplate$$FastClassBySpringCGLIB$$9cb7880b.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.budo.graph.druid.spring.aop.interceptor.BudoGraphDruidSpringAopInterceptor.invoke(BudoGraphDruidSpringAopInterceptor.java:58)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
    at com.ewei.support.spring.hibernate.HibernateTemplate$$EnhancerBySpringCGLIB$$37b9f0f6.executeWithNativeSession(<generated>)
    at com.ewei.support.dao.hibernate.HibernateDao.findByHql(HibernateDao.java:242)
    at com.ewei.support.dao.hibernate.AbstractDao.findByHql(AbstractDao.java:82)
    at com.ewei.helpcenter.dao.impl.HelpCenterDaoImpl.findByProviderId(HelpCenterDaoImpl.java:39)
    at com.ewei.helpcenter.dao.impl.HelpCenterDaoImpl$$FastClassBySpringCGLIB$$6ed9eff7.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.budo.graph.druid.spring.aop.interceptor.BudoGraphDruidSpringAopInterceptor.invoke(BudoGraphDruidSpringAopInterceptor.java:58)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
    at com.ewei.helpcenter.dao.impl.HelpCenterDaoImpl$$EnhancerBySpringCGLIB$$9ed95b34.findByProviderId(<generated>)
    at com.ewei.helpcenter.service.impl.HelpCenterServiceImpl.findByProviderId(HelpCenterServiceImpl.java:101)
    at com.ewei.helpcenter.service.impl.HelpCenterServiceImpl$$FastClassBySpringCGLIB$$43f95f97.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.budo.graph.druid.spring.aop.interceptor.BudoGraphDruidSpringAopInterceptor.invoke(BudoGraphDruidSpringAopInterceptor.java:58)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:185)
    at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:60)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
    at com.ewei.helpcenter.service.impl.HelpCenterServiceImpl$$EnhancerBySpringCGLIB$$38be7f2c.findByProviderId(<generated>)
    at com.alibaba.dubbo.common.bytecode.Wrapper16.invokeMethod(Wrapper16.java)
    at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:45)
    at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:71)
    at com.alibaba.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:48)
    at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:52)
    at org.budo.dubbo.protocol.http.exception.filter.BudoExceptionFilter.invoke(BudoExceptionFilter.java:45)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.ewei.support.dubbo.filter.entitywriteback.ProviderSideEntityWriteBackFilter.invoke(ProviderSideEntityWriteBackFilter.java:26)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.ewei.support.dubbo.filter.anchor.ProviderSideLoggerAnchorFilter.invoke(ProviderSideLoggerAnchorFilter.java:27)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:41)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:77)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at org.budo.graph.druid.dubbo.filter.BudoGraphDruidDubboProviderSideFilter.invoke(BudoGraphDruidDubboProviderSideFilter.java:88)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:74)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.ewei.common.tlog.EweiTLogDubboFilter.invoke(EweiTLogDubboFilter.java:64)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:131)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:37)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:37)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:68)
    at org.budo.dubbo.protocol.dubbo.AbstractBudoDubboProtocol$1.reply(AbstractBudoDubboProtocol.java:100)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:96)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:168)
    at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:50)
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:79)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.ewei.account.api.entity.Provider#7893]
    at org.hibernate.internal.SessionFactoryImpl$1$1.handleEntityNotFound(SessionFactoryImpl.java:253)
    at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:218)
    at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:274)
    at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:150)
    at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070)
    at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:989)
    at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:716)
    at org.hibernate.type.EntityType.resolve(EntityType.java:502)
    at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:170)
    at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:144)
    at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.performTwoPhaseLoad(AbstractRowReader.java:244)
    at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.finishUp(AbstractRowReader.java:215)
    at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:140)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:138)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:102)
    at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.loadEntityBatch(AbstractLoadPlanBasedEntityLoader.java:145)
    at org.hibernate.loader.entity.plan.LegacyBatchingEntityLoaderBuilder$LegacyBatchingEntityLoader.load(LegacyBatchingEntityLoaderBuilder.java:116)
    at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4126)
    at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:502)
    at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:467)
    at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:212)
    at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:274)
    at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:150)
    at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070)
    at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:989)
    at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:716)
    at org.hibernate.type.EntityType.resolve(EntityType.java:502)
    at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:170)
    at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:144)
    at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1114)
    at org.hibernate.loader.Loader.processResultSet(Loader.java:972)
    at org.hibernate.loader.Loader.doQuery(Loader.java:920)
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:354)
    at org.hibernate.loader.Loader.doList(Loader.java:2553)
    at org.hibernate.loader.Loader.doList(Loader.java:2539)
    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2369)
    at org.hibernate.loader.Loader.list(Loader.java:2364)
    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:496)
    at com.ewei.support.hibernate.hql.internal.ast.AbstractQueryTranslator.list(AbstractQueryTranslator.java:325)
    at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:231)
    at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1264)
    at org.hibernate.internal.QueryImpl.list(QueryImpl.java:103)
    at org.hibernate.internal.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:966)
    at com.ewei.support.hibernate.wrapper.QueryWrapper.uniqueResult(QueryWrapper.java:160)
    at com.ewei.support.dao.hibernate.HibernateDao$3.doInHibernate(HibernateDao.java:248)
    at org.springframework.orm.hibernate4.HibernateTemplate.doExecute(HibernateTemplate.java:340)
    ... 73 more

1.因为是hibernate这种错误一般是级联查询没有查询到数据导致的,我们去数据库查询数据发现有数据

3.怀疑是hibernate级联没查询到数据或者底层发sql到数据库服务出现了啥问题

如:

1.最终发送sql的参数不一致

2.查询到数据，在框架内部处理过程中resultSet被读了。或者读出数据处理过程中丢掉了

级联触发位置

org.hibernate.engine.internal.TwoPhaseLoad#doInitializeEntity


#级联复用连接的地方
org.hibernate.engine.jdbc.internal.LogicalConnectionImpl#getConnection

#代理的statement
com.alibaba.druid.pool.DruidPooledPreparedStatement#DruidPooledPreparedStatement

#执行statement的地方
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl#extract(java.sql.PreparedStatement)

#最终statement
com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl#executeQuery
->
com.alibaba.druid.filter.FilterChainImpl#preparedStatement_executeQuery

4.跟源码跟到最终发送sql的statement sql和数据都是一致,通过Arthas增加日志发现确实没有查询到数据

准备尝试使用tcp抓包看发给mysql 的包里面sql和参数正常不

5.后面发现老版本和新版本驱动不一致

非spring boot看lib spring boot项目解压jar包看lib依赖