spring gateway LEAK: ByteBuf.release() was not called before it's garbage-collected

公司外接的项目上线一个多月，项目组说：网关突然报错了，拿来日志如下，原来是堆外内存不够用了？！

 1 2023-06-19 15:55:28.740 [reactor-http-epoll-3] [7529972426645015] [ERROR] reactor.util.Loggers$Slf4JLogger:319 [error] - Operator called default onErrorDropped
 2 reactor.netty.ReactorNetty$InternalNettyException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4096 byte(s) of direct memory (used: 67106823, max: 67108864)
 3 Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4096 byte(s) of direct memory (used: 67106823, max: 67108864)
 4     at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:776)
 5     at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:731)
 6     at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
 7     at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64)
 8     at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41)
 9     at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
10     at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:632)
11     at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:381)
12     at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
13     at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
14     at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
15     at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
16     at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
17     at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780)
18     at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
19     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
20     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
21     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
22     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
23     at java.lang.Thread.run(Thread.java:750)

SpringCloudGateway 是基于 WebFlux 框架实现的，而 WebFlux 框架底层则使用了高性能的 Reactor 模式通信框架 Netty，堆外加内存不够用？打开netty的采样器并输出详细的日志，和生成dump文件

先看一下环境的框架版本：

jdk1.8
springframework——5.2.13.RELEASE
spring-boot——2.3.9.RELEASE
spring-cloud——Hoxton.SR8
nacos-client——1.4.2
spring-cloud-alibaba——2.2.5.RELEASE

再了解一下线上网关的相关参数
小内存运行一下，并打开netty的采样器并输出详细的日志，和生成dump文件

-Xms128m
-Xmx128m
-Xmn16m
-Xss2m
-XX:MetaspaceSize=64m
-XX:MaxMetaspaceSize=64m
-XX:ParallelGCThreads=20
-XX:MaxDirectMemorySize=64m
-XX:MaxDirectMemorySize=64M
-XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/Volumes/Netsc/XXX/springcloud-gateway/heapDump.hprof
-Dio.netty.leakDetection.level=advanced
-Dio.netty.leakDetectionLevel=advanced
-Dio.netty.leakDetection.targetRecords=10

日志如下，还是不明显

2023-05-19 11:26:00.096 [reactor-http-epoll-7] [7664491715816134] [ERROR] io.netty.util.ResourceLeakDetector:319 [reportTracedLeak] - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
    io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
    io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:336)
    io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:388)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:750)

关闭fastThrow，打开内存的详细日志，Jmeter压测

-XX:-OmitStackTraceInFastThrow
-XX:NativeMemoryTracking=detail

日志如下：

2023-07-11 14:27:19.282 ERROR 81189 --- [ctor-http-nio-3] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    io.netty.buffer.AdvancedLeakAwareByteBuf.nioBuffer(AdvancedLeakAwareByteBuf.java:712)
    org.springframework.core.io.buffer.NettyDataBuffer.asByteBuffer(NettyDataBuffer.java:291)
    org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:200)
    org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:63)
    org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:114)
    org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:102)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107)
    reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)
    reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
    reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330)
    reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812)
    reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152)
    reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1813)
    reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
    reactor.core.publisher.Mono.subscribe(Mono.java:4252)
    reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
    reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
    reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.Mono.subscribe(Mono.java:4252)
    reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
    reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
    reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
    reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:93)
    reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)
    reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
    reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330)
    reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812)
    reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152)
    reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
    reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252)
    reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
    reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:450)
    reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:256)
    reactor.netty.channel.FluxReceive.request(FluxReceive.java:130)
    reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:155)
    reactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:130)
    reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:155)
    reactor.core.publisher.MonoCollect$CollectSubscriber.onSubscribe(MonoCollect.java:96)
    reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:86)
    reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:163)
    reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:86)
    reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:168)
    reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:147)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)
#2:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
    io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:616)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:258)
    io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)
#3:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
    io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:596)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:258)
    io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)
#4:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
    io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:997)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:239)
    io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:123)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)
#5:
    Hint: 'reactor.left.httpCodec' will handle the message from this point.
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)
#6:
    Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)
#7:
    io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
    io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:385)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
    io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:745)

没有公司自己的包名出现，然而 netty 的问题不太大，主要是spring的相关的报错有几行非常扎眼😉

    org.springframework.core.io.buffer.NettyDataBuffer.asByteBuffer(NettyDataBuffer.java:291)
    org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:200)
    org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:63)
    org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:114)
    org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:102)

点进去看看源码，到底发生了什么？
1. ε=(´ο｀*)))唉，有堆外内存的释放相关操作，但是到底释放成功了吗？可以的debugger一下看看　　
果然是没有释放成功，debug 显示内存计数器没有归零，故，回收失败！
上网搜索🔍相关资料：spring gateway 相关版本果然有堆外内存释放相关问题！
好吧。不想升级框架。尝试修改源码试试
1. 此方法是最后一个使用这个 DataBuffer 的使用者了，之后就 decode 成 String 了，因此，要在此方法里要彻底释放内存！源码如下：
2. 　
maven 打包，测试，Jmeter 压力测试2小时没有问题，问题暂定已解决　　
合代码，打包，上测试环境，重启网关服务。压力测试12小时。
搞定！下班！明天来了，see see日志，查找关键字【leak】和【exception】看看
活力满满地来上班了，看看禅道和OA等，再上测试环境看看日志。
没有问题，小内存测试通过！今天再用和正常环境一样的内存参数启动服务，再压力测试到下班 ——8小时！
下班！明天来了，see see日志，查找关键字【leak】和【exception】看看
活力满满地来上班了，看看禅道和OA等，再上测试环境看看日志，查找关键字【leak】和【exception】看看。
没有问题，测试通过！提交代码，合代码，让 Jenkins 打个包，发个版，k8s 部署一下。上压测环境，高压测试一下，又是8小时！
没有问题，测试通过！搞定。

posted @ 2023-07-11 17:10 望)=天涯阅读(841) 评论(0) 编辑收藏举报

刷新页面返回顶部

天涯

spring gateway LEAK: ByteBuf.release() was not called before it's garbage-collected

公告