spring gateway LEAK: ByteBuf.release() was not called before it's garbage-collected
- 公司外接的项目上线一个多月,项目组 说:网关突然报错了,拿来日志如下,原来是堆外内存不够用了?!
1 2023-06-19 15:55:28.740 [reactor-http-epoll-3] [7529972426645015] [ERROR] reactor.util.Loggers$Slf4JLogger:319 [error] - Operator called default onErrorDropped 2 reactor.netty.ReactorNetty$InternalNettyException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4096 byte(s) of direct memory (used: 67106823, max: 67108864) 3 Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4096 byte(s) of direct memory (used: 67106823, max: 67108864) 4 at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:776) 5 at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:731) 6 at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) 7 at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) 8 at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) 9 at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) 10 at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:632) 11 at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:381) 12 at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) 13 at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) 14 at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) 15 at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) 16 at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) 17 at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780) 18 at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) 19 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) 20 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) 21 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 22 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 23 at java.lang.Thread.run(Thread.java:750)
- SpringCloudGateway 是基于
WebFlux
框架实现的,而WebFlux
框架底层则使用了高性能的Reactor
模式通信框架 Netty,堆外加内存不够用?打开netty的采样器并输出详细的日志,和生成dump文件- 先看一下环境的框架版本:
-
jdk1.8
springframework——5.2.13.RELEASE spring-boot——2.3.9.RELEASE spring-cloud——Hoxton.SR8 nacos-client——1.4.2 spring-cloud-alibaba——2.2.5.RELEASE
-
- 再了解一下线上网关的相关参数
- 小内存运行一下,并打开netty的采样器并输出详细的日志,和生成dump文件
-
-Xms128m -Xmx128m -Xmn16m -Xss2m -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:ParallelGCThreads=20 -XX:MaxDirectMemorySize=64m -XX:MaxDirectMemorySize=64M
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Volumes/Netsc/XXX/springcloud-gateway/heapDump.hprof -Dio.netty.leakDetection.level=advanced -Dio.netty.leakDetectionLevel=advanced -Dio.netty.leakDetection.targetRecords=10
- 先看一下环境的框架版本:
- 日志如下,还是不明显
-
2023-05-19 11:26:00.096 [reactor-http-epoll-7] [7664491715816134] [ERROR] io.netty.util.ResourceLeakDetector:319 [reportTracedLeak] - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143) io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:336) io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:388) io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750)
-
- 关闭fastThrow,打开内存的详细日志,Jmeter压测
-
-XX:-OmitStackTraceInFastThrow -XX:NativeMemoryTracking=detail
-
- 日志如下:
-
2023-07-11 14:27:19.282 ERROR 81189 --- [ctor-http-nio-3] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: io.netty.buffer.AdvancedLeakAwareByteBuf.nioBuffer(AdvancedLeakAwareByteBuf.java:712) org.springframework.core.io.buffer.NettyDataBuffer.asByteBuffer(NettyDataBuffer.java:291) org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:200) org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:63) org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:114) org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:102) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812) reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1813) reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49) reactor.core.publisher.Mono.subscribe(Mono.java:4252) reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) reactor.core.publisher.Mono.subscribe(Mono.java:4252) reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:93) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812) reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:450) reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:256) reactor.netty.channel.FluxReceive.request(FluxReceive.java:130) reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:155) reactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:130) reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:155) reactor.core.publisher.MonoCollect$CollectSubscriber.onSubscribe(MonoCollect.java:96) reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:86) reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:163) reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:86) reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:168) reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:147) io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164) io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745) #2: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940) io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:616) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:258) io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940) io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:596) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:258) io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745) #4: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940) io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:997) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:239) io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745) #5: Hint: 'reactor.left.httpCodec' will handle the message from this point. io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745) #6: Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point. io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745) #7: io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634) io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745) Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:385) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139) io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:745)
-
- 没有公司自己的包名出现,然而 netty 的问题不太大,主要是spring的相关的报错有几行非常扎眼😉
-
org.springframework.core.io.buffer.NettyDataBuffer.asByteBuffer(NettyDataBuffer.java:291) org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:200) org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:63) org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:114) org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:102)
-
- 点进去看看源码,到底发生了什么?
- ε=(´ο`*)))唉,有堆外内存的释放相关操作,但是到底释放成功了吗?可以的debugger一下看看
- 果然是没有释放成功,debug 显示 内存计数器没有归零,故,回收失败!
- 上网搜索🔍相关资料:spring gateway 相关版本果然有堆外内存释放相关问题!
- 好吧。不想升级框架。尝试修改源码试试
- 此方法是最后一个使用这个 DataBuffer 的使用者了,之后就 decode 成 String 了,因此,要在此方法里要彻底释放内存!源码如下:
- maven 打包,测试,Jmeter 压力测试2小时没有问题,问题暂定已解决
- 合代码,打包,上测试环境,重启网关服务。压力测试12小时。
- 搞定!下班!明天来了,see see日志,查找关键字 【leak】和【exception】看看
- 活力满满地来上班了,看看禅道和OA等,再上测试环境看看日志。
- 没有问题,小内存测试通过!今天再用和正常环境一样的内存参数启动服务,再压力测试 到下班 ——8小时!
- 下班!明天来了,see see日志,查找关键字 【leak】和【exception】看看
- 活力满满地来上班了,看看禅道和OA等,再上测试环境看看日志,查找关键字 【leak】和【exception】看看。
- 没有问题,测试通过!提交代码,合代码,让 Jenkins 打个包,发个版,k8s 部署一下。上压测环境,高压测试一下,又是8小时!
- 没有问题,测试通过!搞定。