spring gateway LEAK: ByteBuf.release() was not called before it's garbage-collected

 

  1. 公司外接的项目上线一个多月,项目组 说:网关突然报错了,拿来日志如下,原来是堆外内存不够用了?!
 1 2023-06-19 15:55:28.740 [reactor-http-epoll-3] [7529972426645015] [ERROR] reactor.util.Loggers$Slf4JLogger:319 [error] - Operator called default onErrorDropped
 2 reactor.netty.ReactorNetty$InternalNettyException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4096 byte(s) of direct memory (used: 67106823, max: 67108864)
 3 Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4096 byte(s) of direct memory (used: 67106823, max: 67108864)
 4     at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:776)
 5     at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:731)
 6     at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
 7     at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64)
 8     at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41)
 9     at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
10     at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:632)
11     at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:381)
12     at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
13     at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
14     at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
15     at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
16     at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
17     at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780)
18     at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
19     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
20     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
21     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
22     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
23     at java.lang.Thread.run(Thread.java:750)
  1. SpringCloudGateway 是基于 WebFlux 框架实现的,而 WebFlux 框架底层则使用了高性能的 Reactor 模式通信框架 Netty,堆外加内存不够用?打开netty的采样器并输出详细的日志,和生成dump文件
    1. 先看一下环境的框架版本:
      1.   
        jdk1.8
        springframework——5.2.13.RELEASE spring-boot——2.3.9.RELEASE spring-cloud——Hoxton.SR8 nacos-client——1.4.2 spring-cloud-alibaba——2.2.5.RELEASE
    2. 再了解一下线上网关的相关参数
    3. 小内存运行一下,并打开netty的采样器并输出详细的日志,和生成dump文件
    4. -Xms128m
      -Xmx128m
      -Xmn16m
      -Xss2m
      -XX:MetaspaceSize=64m
      -XX:MaxMetaspaceSize=64m
      -XX:ParallelGCThreads=20
      -XX:MaxDirectMemorySize=64m
      -XX:MaxDirectMemorySize=64M
      -XX:+HeapDumpOnOutOfMemoryError 
      -XX:HeapDumpPath=/Volumes/Netsc/XXX/springcloud-gateway/heapDump.hprof -Dio.netty.leakDetection.level=advanced -Dio.netty.leakDetectionLevel=advanced -Dio.netty.leakDetection.targetRecords=10
  2. 日志如下,还是不明显
    1. 2023-05-19 11:26:00.096 [reactor-http-epoll-7] [7664491715816134] [ERROR] io.netty.util.ResourceLeakDetector:319 [reportTracedLeak] - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
      Recent access records: 
      Created at:
          io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
          io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
          io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:336)
          io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
          io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
          io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
          io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
          io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
          io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
          io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
          io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:388)
          io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
          io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
          io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:750)
        
  3. 关闭fastThrow,打开内存的详细日志,Jmeter压测
    1.   
      -XX:-OmitStackTraceInFastThrow
      -XX:NativeMemoryTracking=detail

       

  4. 日志如下:
    1.   
      2023-07-11 14:27:19.282 ERROR 81189 --- [ctor-http-nio-3] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
      Recent access records: 
      #1:
          io.netty.buffer.AdvancedLeakAwareByteBuf.nioBuffer(AdvancedLeakAwareByteBuf.java:712)
          org.springframework.core.io.buffer.NettyDataBuffer.asByteBuffer(NettyDataBuffer.java:291)
          org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:200)
          org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:63)
          org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:114)
          org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:102)
          reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107)
          reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)
          reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
          reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330)
          reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812)
          reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152)
          reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1813)
          reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
          reactor.core.publisher.Mono.subscribe(Mono.java:4252)
          reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
          reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
          reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
          reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
          reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
          reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
          reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
          reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
          reactor.core.publisher.Mono.subscribe(Mono.java:4252)
          reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
          reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
          reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
          reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
          reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
          reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
          reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
          reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:93)
          reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)
          reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
          reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330)
          reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812)
          reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152)
          reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
          reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252)
          reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
          reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:450)
          reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:256)
          reactor.netty.channel.FluxReceive.request(FluxReceive.java:130)
          reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:155)
          reactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:130)
          reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:155)
          reactor.core.publisher.MonoCollect$CollectSubscriber.onSubscribe(MonoCollect.java:96)
          reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:86)
          reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:163)
          reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:86)
          reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:168)
          reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:147)
          io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
          io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
          io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)
      #2:
          io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
          io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
          io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:616)
          io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:258)
          io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
          io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
          io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
          io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
          io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
          io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
          io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
          io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
          io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
          io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)
      #3:
          io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
          io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
          io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:596)
          io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:258)
          io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
          io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
          io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
          io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
          io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
          io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
          io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
          io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
          io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
          io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)
      #4:
          io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
          io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
          io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:997)
          io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:239)
          io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
          io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
          io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
          io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
          io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
          io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
          io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
          io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
          io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
          io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)
      #5:
          Hint: 'reactor.left.httpCodec' will handle the message from this point.
          io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
          io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
          io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
          io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
          io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
          io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
          io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
          io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)
      #6:
          Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
          io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
          io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
          io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
          io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
          io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
          io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
          io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)
      #7:
          io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
          io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
          io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
          io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
          io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
          io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)
      Created at:
          io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:385)
          io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
          io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
          io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
          io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
          io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
          io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
          io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
          io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
          io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
          io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
          io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          java.lang.Thread.run(Thread.java:745)

       

  5. 没有公司自己的包名出现,然而 netty 的问题不太大,主要是spring的相关的报错有几行非常扎眼😉
    1.   
          org.springframework.core.io.buffer.NettyDataBuffer.asByteBuffer(NettyDataBuffer.java:291)
          org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:200)
          org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:63)
          org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:114)
          org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:102)
          
  6. 点进去看看源码,到底发生了什么?
    1. ε=(´ο`*)))唉,有堆外内存的释放相关操作,但是到底释放成功了吗?可以的debugger一下看看  
  7. 果然是没有释放成功,debug 显示 内存计数器没有归零,故,回收失败!
  8. 上网搜索🔍相关资料:spring gateway 相关版本果然有堆外内存释放相关问题!
  9. 好吧。不想升级框架。尝试修改源码试试
    1. 此方法是最后一个使用这个 DataBuffer 的使用者了,之后就 decode 成 String 了,因此,要在此方法里要彻底释放内存!源码如下:
    2.  
  10. maven 打包,测试,Jmeter 压力测试2小时没有问题,问题暂定已解决  
  11. 合代码,打包,上测试环境,重启网关服务。压力测试12小时。
  12. 搞定!下班!明天来了,see see日志,查找关键字 【leak】和【exception】看看



  13. 活力满满地来上班了,看看禅道和OA等,再上测试环境看看日志。
  14. 没有问题,小内存测试通过!今天再用和正常环境一样的内存参数启动服务,再压力测试 到下班 ——8小时!
  15. 下班!明天来了,see see日志,查找关键字 【leak】和【exception】看看



  16. 活力满满地来上班了,看看禅道和OA等,再上测试环境看看日志,查找关键字 【leak】和【exception】看看。
  17. 没有问题,测试通过!提交代码,合代码,让 Jenkins 打个包,发个版,k8s 部署一下。上压测环境,高压测试一下,又是8小时!
  18. 没有问题,测试通过!搞定。

 

posted @ 2023-07-11 17:10  望)=天涯  阅读(687)  评论(0编辑  收藏  举报