记一次 公司.NET项目部署在Linux环境压测时 内存暴涨分析
一：背景
- 讲故事
公司部署在某碟上的项目在9月份压测50并发时，发现某个容器线程、内存非正常的上涨，导致功能出现了异常无法使用。根据所学，自己分析了下线程和内存问题，分析时可以使用lldb或者windbg，但是个人比较倾向于界面化的windbg，所以最终使用windbg开干。
二：WinDbg 分析
- 到底是哪里的泄漏
在 windows 平台上相信有很多朋友都知道用 !address -summary 命令看，但这是专属于windows平台的命令，在分析linux上的dump不好使，参考如下输出：
0:000> !address -summary
Mapping file section regions...
Mapping module regions...
Mapping heap regions...
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
<unknown> 4062 ffffffff`f5638600 ( 16.000 EB) 100.00% 100.00%
Image 1282 0`09fc8a00 ( 159.784 MB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
2431 fffffffe`2b813000 ( 16.000 EB) 100.00%
MEM_PRIVATE 2913 1`d3dee000 ( 7.310 GB) 0.00% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
2431 fffffffe`2b813000 ( 16.000 EB) 100.00% 100.00%
MEM_COMMIT 2913 1`d3dee000 ( 7.310 GB) 0.00% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 2115 1`cb683000 ( 7.178 GB) 0.00% 0.00%
PAGE_EXECUTE_READ 175 0`03d49000 ( 61.285 MB) 0.00% 0.00%
PAGE_READONLY 585 0`03ce9000 ( 60.910 MB) 0.00% 0.00%
PAGE_EXECUTE_WRITECOPY 38 0`00d39000 ( 13.223 MB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unknown> 7ffc`011fa000 ffff8003`fe406000 ( 16.000 EB)
Image 7f45`fe4e9000 0`01b16000 ( 27.086 MB)
卦中的内存段分类用处不大，也没有多大的参考价值，那怎么办呢？其实 coreclr 团队也考虑到了这个情况，它提供了一个 maddress 命令来实现跨平台的 !address，更改后输出如下：
0:000> !sos maddress
Enumerating and tagging the entire address space and caching the result...
Subsequent runs of this command should be faster.
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Memory Kind | StartAddr | EndAddr-1 | Size | Type | State | Protect | Image |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Stack | 7f42d256e000 | 7f42d2d6e000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d3570000 | 7f42d3d70000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d3d71000 | 7f42d4571000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d4572000 | 7f42d4d72000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d4d73000 | 7f42d5573000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d5574000 | 7f42d5d74000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d5d75000 | 7f42d6575000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d6d77000 | 7f42d7577000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d7578000 | 7f42d7d78000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d7d79000 | 7f42d8579000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7f42d857a000 | 7f42d8d7a000 | 8.00mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
...
+-------------------------------------------------------------------------+
| Memory Type | Count | Size | Size (bytes) |
+-------------------------------------------------------------------------+
| Stack | 788 | 6.28gb | 6,743,269,376 |
| GCHeap | 48 | 688.98mb | 722,448,384 |
| PAGE_READWRITE | 930 | 180.22mb | 188,977,152 |
| Image | 1,278 | 159.69mb | 167,447,040 |
| HighFrequencyHeap | 327 | 20.35mb | 21,336,064 |
| LowFrequencyHeap | 259 | 18.31mb | 19,202,048 |
| LoaderCodeHeap | 15 | 17.53mb | 18,378,752 |
| HostCodeHeap | 11 | 1.51mb | 1,581,056 |
| ResolveHeap | 1 | 348.00kb | 356,352 |
| PAGE_READONLY | 123 | 261.50kb | 267,776 |
| DispatchHeap | 1 | 196.00kb | 200,704 |
| IndirectionCellHeap | 3 | 152.00kb | 155,648 |
| LookupHeap | 3 | 144.00kb | 147,456 |
| CacheEntryHeap | 2 | 100.00kb | 102,400 |
| PAGE_EXECUTE_WRITECOPY | 5 | 96.00kb | 98,304 |
| StubHeap | 2 | 76.00kb | 77,824 |
| PAGE_EXECUTE_READ | 2 | 8.00kb | 8,192 |
+-------------------------------------------------------------------------+
| [TOTAL] | 3,798 | 7.34gb | 7,884,054,528 |
+-------------------------------------------------------------------------+
从卦中可以看到当前程序总计 6.28gb 内存占用，基本上都被线程栈给吃掉了，更让人意想不到的是这个线程栈居然占用 8M 的内存空间，这个着实有点大了，而且 linux 不像 windows 有一个 reserved 的概念，这里的 8M 是实实在在的预占，可以观察这 8M 的内存地址即可，都是初始化的 0, 这就说不过去了。
0:000> dp 7f42d256e000 7f42d2d6e000
...
00007f42`d2d6dfa0 00000000`00000000 00000000`00000000
00007f42`d2d6dfb0 00000000`00000000 00000000`00000000
00007f42`d2d6dfc0 00000000`00000000 00000000`00000000
00007f42`d2d6dfd0 00000000`00000000 00000000`00000000
00007f42`d2d6dfe0 00000000`00000000 00000000`00000000
00007f42`d2d6dff0 00000000`00000000 00000000`00000000
00007f42`d2d6e000 ????????`????????
- 如何修改栈空间大小
一般来说不同的操作系统发行版有不同的默认栈空间配置，可以先到内存搜一下当前是哪一个发行版，做法就是搜索操作系统名称主要关键字。
0:000> s-a 0 L?0xffffffffffffffff "centos"
...
00005570`9cddbc18 63 65 6e 74 6f 73 2e 37-2d 78 36 34 00 00 00 00 centos.7-x64....
...
从卦中可以看到当前操作系统是 centos7-x64，在 windows 平台上修改栈空间大小可以修改 PE 头，在 linux 上有两种做法。
修改 ulimit -s 参数（不建议）
root@ubuntu:/data# ulimit -s
8192
root@ubuntu:/data# ulimit -s 2048
root@ubuntu:/data# ulimit -s
2048
修改 DOTNET_DefaultStackSize 环境变量（建议，针对异常容器在环境变量配置）
DOTNET_DefaultStackSize=180000
更多可以参考文章： https://www.alexander-koepke.de/post/2023-10-18-til-dotnet-stack-size/
上面是解决问题的第一个方向，接下来我们说另一个方向，为什么会产生总计 788 个线程呢？
- 为什么会有那么多线程
要找到这个答案，需要去看每一个线程此时都在干嘛，这个可以使用 windbg 专属命令。
0:000> ~*e !clrstack
...
OS Thread Id: 0x1b82 (225)
Child SP IP Call Site
00007F441B7FD660 00007f4cdbb69ad8 [HelperMethodFrame_1OBJ: 00007f441b7fd660] System.Threading.Monitor.ObjWait(Int32, System.Object)
00007F441B7FD790 00007f4c676318cd System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ManualResetEventSlim.cs @ 570]
00007F441B7FD810 00007f4c676312e1 System.Net.Sockets.SocketAsyncContext.PerformSyncOperation[[System.__Canon, System.Private.CoreLib]](OperationQueue`1<System.__Canon> ByRef, System.__Canon, Int32, Int32) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 1330]
00007F441B7FD8A0 00007f4c67e26ff1 System.Net.Sockets.SocketAsyncContext.ReceiveFrom(System.Memory`1, System.Net.Sockets.SocketFlags ByRef, Byte[], Int32 ByRef, Int32, Int32 ByRef) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 1557]
00007F441B7FD920 00007f4c67e2ea6b System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle, Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, Int32 ByRef)
00007F441B7FD9A0 00007f4c67e26c37 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)
00007F441B7FDA20 00007f4c67e26929 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs @ 231]
00007F441B7FDA70 00007f4c69b85757 System.IO.BufferedStream.ReadByteSlow() [/_/src/libraries/System.Private.CoreLib/src/System/IO/BufferedStream.cs @ 771]
00007F441B7FDA90 00007f4c69b774e8 System.IO.BinaryReader.ReadByte() [/_/src/libraries/System.Private.CoreLib/src/System/IO/BinaryReader.cs @ 207]
00007F441B7FDAA0 00007f4c69b853ee RabbitMQ.Client.Impl.InboundFrame.ReadFrom(RabbitMQ.Util.NetworkBinaryReader)
00007F441B7FDAF0 00007f4c69b852c6 RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
00007F441B7FDB10 00007f4c69b57068 RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
00007F441B7FDB50 00007f4c67590d19 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
00007F441B7FDCF0 00007f4cdb1e3aa7 [DebuggerU2MCatchHandlerFrame: 00007f441b7fdcf0]
...
可以使用正规的 dotnet-dump 或者 procdump抓取，根据上面卦象展示，可以看到大量的和 RabbitMQ.Client.Framing.Impl 有关的链接库，猜测大量线程卡在 RabbitMQ.Client.Framing.Impl 中。
有了这些知识，最后给到朋友的建议如下：
修改 DOTNET_DefaultStackSize 参数
可以仿照 windows 上的 .netcore 默认 1.5M 的栈空间设置，因为8M真的太大了，扛不住，也和 Linux 的低内存使用不符。修改后压测读取dump观察发现配置已生效
0:000> !sos maddress
Enumerating and tagging the entire address space and caching the result...
Subsequent runs of this command should be faster.
*** WARNING: Unable to verify timestamp for lttng-ust-wait-8-0
*** WARNING: Unable to verify timestamp for lttng-ust-wait-8
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Memory Kind | StartAddr | EndAddr-1 | Size | Type | State | Protect | Image |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
.......
| Stack | 7fabe4e8c000 | 7fabe500c000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe500d000 | 7fabe518d000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe518e000 | 7fabe530e000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe530f000 | 7fabe548f000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe5490000 | 7fabe5610000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe5611000 | 7fabe5791000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe5792000 | 7fabe5912000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe5913000 | 7fabe5a93000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe5a94000 | 7fabe5c14000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
| Stack | 7fabe5c15000 | 7fabe5d95000 | 1.50mb | MEM_PRIVATE | MEM_COMMIT | PAGE_READWRITE | |
.......
+-------------------------------------------------------------------------+
| Memory Type | Count | Size | Size (bytes) |
+-------------------------------------------------------------------------+
| Stack | 766 | 1.41gb | 1,518,571,520 |
| GCHeap | 48 | 702.39mb | 736,509,952 |
| PAGE_READWRITE | 931 | 186.31mb | 195,358,720 |
| Image | 1,283 | 158.77mb | 166,480,384 |
| HighFrequencyHeap | 336 | 20.97mb | 21,991,424 |
| LowFrequencyHeap | 256 | 18.32mb | 19,214,336 |
| LoaderCodeHeap | 15 | 17.53mb | 18,378,752 |
| HostCodeHeap | 11 | 1.63mb | 1,703,936 |
| ResolveHeap | 1 | 348.00kb | 356,352 |
| PAGE_READONLY | 123 | 261.50kb | 267,776 |
| DispatchHeap | 1 | 196.00kb | 200,704 |
| IndirectionCellHeap | 3 | 152.00kb | 155,648 |
| LookupHeap | 3 | 144.00kb | 147,456 |
| PAGE_EXECUTE_WRITECOPY | 5 | 132.00kb | 135,168 |
| CacheEntryHeap | 2 | 100.00kb | 102,400 |
| StubHeap | 2 | 76.00kb | 77,824 |
| PAGE_EXECUTE_READ | 2 | 8.00kb | 8,192 |
+-------------------------------------------------------------------------+
| [TOTAL] | 3,788 | 2.50gb | 2,679,660,544 |
+-------------------------------------------------------------------------+
观察项目代码中RabbitMQ.Client.Framing.Impl 的相关逻辑
发现该引用其实在代码中属于无效引用，将该引用删除压测观察，发现线程正常。
三：总结
Linux 上的 .NET 调试生态在日渐丰富，这是一件让人很兴奋的事情，最后给我的老师《一线码农》和 WinDbg 点个赞
