记一次 redis 事件注册不当导致的内存泄露

线上的程序跑着跑着内存越来越大,并且没有下降的趋势,重启一下程序也只能短暂恢复。通过 htop 命令再按一下 M 键按内存占用大小排个序,程序会占好几个G。那好,让我们来分析一下。

收集dump

通过 tophtop 进程管理器,或 ps 命令查找到目标进程 id,然后使用如下命令生成 dump:

createdump --full <PID>

dump 会自动保存在 /tmp 目录,文件名格式一般是 coredump.PID

小试牛刀

拿到 dump 后,通过 windbg 打开。首先使用命令 !address -summary 查看一下。

注意:有的朋友可能打开 dump 文件后,发现 sos 扩展加载失败,需要手动加载一下,输入命令 .load sos 即可。

0:000> !address -summary

                                     
Mapping file section regions...
Mapping module regions...
Mapping heap regions...
*** WARNING: Unable to verify timestamp for libc-2.17.so

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
<unknown>                               869 ffffffff`f409b600 (  16.000 EB) 100.00%  100.00%
Image                                  1117        0`0b565a00 ( 181.397 MB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
                                        808 ffffffff`018de000 (  16.000 EB)          100.00%
MEM_PRIVATE                            1178        0`fdd23000 (   3.966 GB)   0.00%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
                                        808 ffffffff`018de000 (  16.000 EB) 100.00%  100.00%
MEM_COMMIT                             1178        0`fdd23000 (   3.966 GB)   0.00%    0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          520        0`f6bf0000 (   3.855 GB)   0.00%    0.00%
PAGE_EXECUTE_READ                       160        0`05141000 (  81.254 MB)   0.00%    0.00%
PAGE_READONLY                           466        0`0147c000 (  20.484 MB)   0.00%    0.00%
PAGE_EXECUTE_WRITECOPY                   32        0`00b76000 (  11.461 MB)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unknown>                              7ffd`7f5ad000 ffff8002`80053000 (  16.000 EB)
Image                                  7f44`7390c000        0`013d2000 (  19.820 MB)

从输出结果中 MEM_COMMIT 一栏可以看出,内存占了3.9G。

下一步要看是托管内存还是非托管内存吃掉。

托管内存

使用 eeheap 命令查看:

0:000> !sos eeheap 
Loader Heap:
----------------------------------------
System Domain:        7f447979a8c0
LoaderAllocator:      7f447979a8c0
LowFrequencyHeap:     7f4406ce0000(10000:c000) 7f4406cc0000(10000:10000) 7f4406ca0000(10000:10000) 7f4406c80000(10000:10000) 7f4406c50000(10000:10000) 7f4406c20000(10000:10000) 7f4406c00000(10000:10000) 7f4406be0000(10000:10000) 7f4406bc0000(10000:10000) 7f4406b60000(10000:10000) 7f4406b30000(10000:10000) 7f4406b00000(10000:10000) 7f4406ad0000(10000:10000) 7f4406a90000(20000:20000) 7f4406a70000(10000:a000) 7f4406a20000(10000:f000) 7f4406a00000(10000:10000) 7f44069e0000(10000:10000) 7f44069d0000(10000:10000) 7f4406990000(40000:40000) 7f4406950000(30000:29000) 7f4406940000(10000:10000) 7f4406720000(10000:f000) 7f4406700000(10000:10000) 7f44066e0000(10000:10000) 7f44066d0000(10000:10000) 7f44066b0000(10000:10000) 7f4406690000(10000:10000) 7f4406670000(10000:10000) 7f4406540000(10000:10000) 7f4406530000(10000:10000) 7f4406390000(10000:b000) 7f4406330000(10000:10000) 7f4406300000(10000:10000) 7f44062e0000(10000:10000) 7f4406270000(10000:10000) 7f4406240000(10000:10000) 7f4406210000(10000:10000) 7f4406200000(10000:10000) 7f44061d0000(30000:2a000) 7f44061b0000(10000:4000) 7f4406190000(10000:10000) 7f4406170000(10000:10000) 7f4406160000(10000:f000) 7f4406140000(10000:c000) 7f4406110000(10000:10000) 7f44060e0000(10000:10000) 7f4405ed0000(10000:10000) 7f4405eb0000(10000:10000) 7f4405e90000(10000:10000) 7f4405e70000(10000:10000) 7f4405e50000(10000:f000) 7f4405d70000(10000:10000) 7f4405d50000(10000:10000) 7f4405d20000(10000:10000) 7f4405d00000(10000:10000) 7f4405ce0000(10000:10000) 7f4405cd0000(10000:10000) 7f4405cb0000(10000:f000) 7f4405c90000(10000:f000) 7f4405bd0000(10000:10000) 7f4405b40000(10000:10000) 7f4405b20000(10000:f000) 7f4405af0000(10000:10000) 7f4405ad0000(10000:10000) 7f4405aa0000(30000:2f000) 7f4405a90000(10000:e000) 7f4405a70000(10000:10000) 7f4405a30000(10000:10000) 7f4405a10000(10000:10000) 7f44059a0000(10000:10000) 7f4405980000(10000:10000) 7f4405970000(10000:10000) 7f4405950000(10000:10000) 7f4405730000(10000:10000) 7f4405720000(10000:10000) 7f4405700000(10000:10000) 7f44056e0000(10000:10000) 7f44056b0000(10000:10000) 7f44056a0000(10000:10000) 7f4405670000(10000:10000) 7f4405650000(10000:10000) 7f4405630000(10000:f000) 7f44055e0000(10000:10000) 7f44055d0000(10000:10000) 7f44055c0000(10000:10000) 7f44055a0000(10000:10000) 7f4405590000(10000:10000) 7f4405580000(10000:10000) 7f4405560000(10000:10000) 7f4405540000(10000:10000) 7f4405520000(10000:10000) 7f4405510000(10000:10000) 7f44054f0000(10000:10000) 7f44054e0000(10000:10000) 7f44054c0000(10000:10000) 7f44054a0000(10000:10000) 7f4405490000(10000:10000) 7f4405280000(10000:d000) 7f4405260000(10000:10000) 7f4405240000(10000:10000) 7f4405230000(10000:10000) 7f4405210000(10000:10000) 7f4405200000(10000:10000) 7f44051f0000(10000:10000) 7f44051a0000(10000:10000) 7f4405180000(10000:10000) 7f4405150000(10000:10000) 7f4405120000(10000:10000) 7f44050e0000(10000:10000) 7f4405050000(10000:10000) 7f4405020000(10000:10000) 7f4404ff0000(10000:10000) 7f4404fc0000(10000:10000) 7f4404fb0000(10000:10000) 7f4404f90000(10000:f000) 7f4404f50000(10000:10000) 7f4404f40000(10000:10000) 7f4404f30000(10000:10000) 7f4404f10000(10000:7000) 7f4404e60000(10000:10000) 7f4404e50000(10000:e000) 7f4404e30000(10000:d000) 7f4404d20000(10000:10000) 7f4404d10000(10000:10000) 7f4404cf0000(10000:7000) 7f4404c70000(10000:10000) 7f4404c50000(10000:10000) 7f4404a40000(10000:b000) 7f4404a20000(10000:e000) 7f4404a00000(10000:10000) 7f4404970000(10000:10000) 7f4404950000(10000:10000) 7f4404940000(10000:10000) 7f4404930000(10000:7000) 7f4404920000(10000:f000) 7f4404900000(10000:10000) 7f44048e0000(10000:10000) 7f4404830000(10000:10000) 7f4404810000(10000:10000) 7f44047f0000(10000:c000) 7f4404710000(10000:10000) 7f44046f0000(10000:10000) 7f44046d0000(10000:10000) 7f44046a0000(10000:10000) 7f4404550000(10000:10000) 7f44044a0000(10000:10000) 7f44043d0000(10000:10000) 7f44043c0000(10000:10000) 7f44040c0000(10000:b000) 7f4404000000(10000:e000) 7f4403ed0000(10000:10000) 7f4403e30000(10000:10000) 7f4403d10000(10000:10000) 7f4403d00000(10000:f000) 7f4403c90000(10000:a000) 7f4403bc0000(10000:10000) 7f4403bb0000(10000:f000) 7f44037a0000(10000:a000) 7f4403770000(10000:10000) 7f4403730000(10000:10000) 7f44036d0000(10000:10000) 7f4403660000(10000:10000) 7f4403510000(10000:10000) 7f44033f0000(10000:c000) 7f44032a0000(10000:10000) 7f4403280000(10000:10000) 7f4403270000(10000:f000) 7f4403220000(40000:3f000) 7f4402980000(20000:1a000) 7f4402650000(10000:d000) 7f4402620000(10000:d000) 7f4402580000(10000:10000) 7f4402360000(10000:f000) 7f4402210000(10000:10000) 7f4402140000(10000:10000) 7f4402050000(10000:10000) 7f4402030000(10000:10000) 7f4401ee0000(10000:10000) 7f4401d70000(10000:10000) 7f4401cb0000(10000:10000) 7f4401ad0000(10000:10000) 7f44018b0000(10000:f000) 7f44016d0000(10000:10000) 7f4401520000(10000:10000) 7f44013b0000(10000:10000) 7f4401210000(20000:1f000) 7f4400f90000(10000:3000) 7f4400de0000(10000:10000) 7f4400d20000(10000:10000) 7f4400b00000(10000:f000) 7f4400a20000(10000:10000) 7f4400870000(10000:10000) 7f4400730000(20000:20000) 7f44004d0000(10000:2000) 7f4400280000(10000:10000) 7f4400200000(10000:e000) 7f44001f0000(10000:9000) 7f4400100000(10000:10000) 7f44000f0000(10000:10000) 7f44000a0000(10000:10000) 7f4400000000(10000:10000) 7f43fffb0000(50000:50000) 7f43ff4e0000(3000:1000) Size: 0xd3b000 (13873152) bytes total, 0xb4000 (737280) bytes wasted.
HighFrequencyHeap:    7f4406cd0000(10000:f000) 7f4406cb0000(10000:10000) 7f4406c70000(10000:10000) 7f4406c60000(10000:10000) 7f4406c40000(10000:10000) 7f4406c30000(10000:10000) 7f4406c10000(10000:10000) 7f4406bf0000(10000:10000) 7f4406bd0000(10000:10000) 7f4406b70000(10000:10000) 7f4406b50000(10000:10000) 7f4406b40000(10000:10000) 7f4406b10000(10000:10000) 7f4406af0000(10000:10000) 7f4406ae0000(10000:10000) 7f4406ac0000(10000:10000) 7f4406ab0000(10000:10000) 7f4406a80000(10000:10000) 7f4406a60000(10000:10000) 7f4406a50000(10000:10000) 7f4406a40000(10000:10000) 7f4406a30000(10000:10000) 7f4406a10000(10000:10000) 7f44069f0000(10000:10000) 7f4406980000(10000:10000) 7f4406730000(10000:10000) 7f4406710000(10000:10000) 7f44066f0000(10000:10000) 7f44066c0000(10000:10000) 7f44066a0000(10000:10000) 7f4406680000(10000:10000) 7f4406620000(10000:10000) 7f4406550000(10000:10000) 7f44063a0000(10000:10000) 7f4406380000(10000:f000) 7f4406320000(10000:10000) 7f4406310000(10000:10000) 7f44062f0000(10000:f000) 7f44062d0000(10000:10000) 7f4406280000(10000:10000) 7f4406230000(10000:f000) 7f44061c0000(10000:10000) 7f44061a0000(10000:10000) 7f4406180000(10000:10000) 7f4406150000(10000:10000) 7f4406130000(10000:10000) 7f4406100000(10000:10000) 7f44060f0000(10000:10000) 7f4405ec0000(10000:10000) 7f4405ea0000(10000:10000) 7f4405e80000(10000:10000) 7f4405e60000(10000:10000) 7f4405e30000(10000:10000) 7f4405d60000(10000:10000) 7f4405d40000(10000:10000) 7f4405d10000(10000:10000) 7f4405cf0000(10000:10000) 7f4405cc0000(10000:10000) 7f4405ca0000(10000:10000) 7f4405c80000(10000:10000) 7f4405b50000(10000:10000) 7f4405b30000(10000:10000) 7f4405b10000(10000:10000) 7f4405b00000(10000:10000) 7f4405ae0000(10000:10000) 7f4405a80000(10000:f000) 7f4405a60000(10000:10000) 7f4405a20000(10000:10000) 7f4405990000(10000:10000) 7f4405960000(10000:10000) 7f4405740000(10000:10000) 7f4405710000(10000:10000) 7f44056f0000(10000:10000) 7f44056d0000(10000:10000) 7f44056c0000(10000:10000) 7f4405680000(10000:10000) 7f4405660000(10000:10000) 7f4405640000(10000:10000) 7f4405610000(10000:10000) 7f4405600000(10000:10000) 7f44055f0000(10000:10000) 7f44055b0000(10000:10000) 7f4405570000(10000:10000) 7f4405530000(10000:10000) 7f4405500000(10000:10000) 7f44054d0000(10000:10000) 7f44054b0000(10000:10000) 7f4405270000(10000:10000) 7f4405250000(10000:10000) 7f4405220000(10000:10000) 7f44051e0000(10000:10000) 7f4405190000(10000:10000) 7f4405170000(10000:10000) 7f4405160000(10000:10000) 7f4405140000(10000:10000) 7f4405130000(10000:10000) 7f4405110000(10000:10000) 7f4405100000(10000:10000) 7f44050f0000(10000:10000) 7f4405060000(10000:10000) 7f4405040000(10000:10000) 7f4405030000(10000:10000) 7f4405010000(10000:10000) 7f4405000000(10000:10000) 7f4404fe0000(10000:10000) 7f4404fd0000(10000:10000) 7f4404fa0000(10000:10000) 7f4404f80000(10000:10000) 7f4404f60000(10000:10000) 7f4404f20000(10000:10000) 7f4404f00000(10000:10000) 7f4404e40000(10000:10000) 7f4404d00000(10000:10000) 7f4404cd0000(10000:10000) 7f4404c60000(10000:10000) 7f4404a30000(10000:10000) 7f4404a10000(10000:10000) 7f4404980000(10000:10000) 7f4404960000(10000:10000) 7f4404910000(10000:10000) 7f44048f0000(10000:10000) 7f44048d0000(10000:10000) 7f44048c0000(10000:10000) 7f4404820000(10000:10000) 7f4404800000(10000:10000) 7f44047e0000(10000:10000) 7f4404700000(10000:10000) 7f44046e0000(10000:10000) 7f44046c0000(10000:10000) 7f44046b0000(10000:10000) 7f4404690000(10000:10000) 7f44045b0000(10000:10000) 7f4404540000(10000:10000) 7f44044b0000(10000:10000) 7f4404490000(10000:10000) 7f4404480000(10000:10000) 7f44043e0000(10000:10000) 7f4404020000(10000:10000) 7f4404010000(10000:10000) 7f4403fa0000(10000:10000) 7f4403ee0000(10000:10000) 7f4403e40000(10000:10000) 7f4403d30000(10000:10000) 7f4403d20000(10000:10000) 7f4403ca0000(10000:10000) 7f4403c70000(10000:10000) 7f4403c60000(10000:10000) 7f4403bd0000(10000:10000) 7f44037b0000(10000:10000) 7f4403790000(10000:10000) 7f4403780000(10000:10000) 7f4403760000(10000:10000) 7f4403750000(10000:10000) 7f4403740000(10000:10000) 7f4403720000(10000:10000) 7f4403710000(10000:10000) 7f44036e0000(10000:10000) 7f44036c0000(10000:10000) 7f44036b0000(10000:10000) 7f4403420000(10000:10000) 7f4403410000(10000:10000) 7f4403400000(10000:10000) 7f44033a0000(10000:10000) 7f4403330000(10000:10000) 7f4403290000(10000:10000) 7f4403260000(10000:10000) 7f4402660000(10000:10000) 7f4402640000(10000:10000) 7f4402630000(10000:10000) 7f4402590000(10000:10000) 7f4402480000(10000:10000) 7f4402370000(10000:10000) 7f44022e0000(10000:10000) 7f4402230000(10000:10000) 7f4402220000(10000:10000) 7f4402130000(10000:10000) 7f4402120000(10000:10000) 7f4402070000(10000:10000) 7f4402060000(10000:10000) 7f4402040000(10000:10000) 7f4402020000(10000:10000) 7f4401df0000(10000:10000) 7f4401d80000(10000:10000) 7f4401d60000(10000:10000) 7f4401d00000(10000:10000) 7f44019c0000(10000:10000) 7f4401850000(10000:10000) 7f44016e0000(10000:10000) 7f4401600000(10000:10000) 7f4401510000(10000:10000) 7f44012e0000(10000:10000) 7f4400f20000(10000:10000) 7f4400d40000(10000:10000) 7f4400d30000(10000:10000) 7f4400b70000(10000:10000) 7f4400a30000(10000:10000) 7f44009c0000(10000:10000) 7f44004e0000(10000:10000) 7f4400110000(10000:10000) 7f44000e0000(10000:10000) 7f44000d0000(10000:10000) 7f44000c0000(10000:10000) 7f44000b0000(10000:10000) 7f4400010000(10000:10000) 7f43ff4e4000(9000:5000) Size: 0xcc0000 (13369344) bytes total, 0x8000 (32768) bytes wasted.
StubHeap:             7f4405070000(10000:5000) 7f43ff4ed000(3000:3000) Size: 0x8000 (32768) bytes total.
IndirectionCellHeap:  7f4405690000(10000:5000) 7f43ff4f0000(6000:6000) Size: 0xb000 (45056) bytes total.
LookupHeap:           7f4404f70000(10000:6000) 7f43ff4ff000(4000:4000) Size: 0xa000 (40960) bytes total.
ResolveHeap:          7f43ff534000(57000:20000) Size: 0x20000 (131072) bytes total.
DispatchHeap:         7f43ff503000(31000:e000) Size: 0xe000 (57344) bytes total.
CacheEntryHeap:       7f4406220000(10000:3000) 7f43ff4f6000(9000:9000) Size: 0xc000 (49152) bytes total.
Total size:           Size: 0x1a52000 (27598848) bytes total, 0xbc000 (770048) bytes wasted.
----------------------------------------
Domain 1:             5558fbe901d0
LoaderAllocator:      5558fbe901d0
No unique loader heaps found.
----------------------------------------
JIT Manager:          5558fbe936c0
LoaderCodeHeap:       7f4400020000(80000:36000) Size: 0x36000 (221184) bytes total.
LoaderCodeHeap:       7f4401940000(80000:39000) Size: 0x39000 (233472) bytes total.
LoaderCodeHeap:       7f44032b0000(80000:42000) Size: 0x42000 (270336) bytes total.
LoaderCodeHeap:       7f4403be0000(80000:49000) Size: 0x49000 (299008) bytes total.
LoaderCodeHeap:       7f44044c0000(80000:5a000) Size: 0x5a000 (368640) bytes total.
LoaderCodeHeap:       7f4404840000(80000:5f000) Size: 0x5f000 (389120) bytes total.
LoaderCodeHeap:       7f4404a50000(200000:18f000) Size: 0x18f000 (1634304) bytes total.
LoaderCodeHeap:       7f4405290000(200000:1c4000) Size: 0x1c4000 (1851392) bytes total.
LoaderCodeHeap:       7f4405750000(200000:1b1000) Size: 0x1b1000 (1773568) bytes total.
LoaderCodeHeap:       7f4405ee0000(200000:1b6000) Size: 0x1b6000 (1794048) bytes total.
LoaderCodeHeap:       7f4406740000(200000:15f000) Size: 0x15f000 (1437696) bytes total.
HostCodeHeap:         7f4403c80000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4404ce0000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4405a50000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4405d80000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4406120000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4406260000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4406290000(40000:40000) Size: 0x40000 (262144) bytes total.
HostCodeHeap:         7f4406340000(40000:40000) Size: 0x40000 (262144) bytes total.
HostCodeHeap:         7f4406b80000(40000:40000) Size: 0x40000 (262144) bytes total.
Total size:           Size: 0xaec000 (11452416) bytes total.
----------------------------------------

========================================
Number of GC Heaps: 1
----------------------------------------
generation 0 starts at 7f423d1f52c8
generation 1 starts at 7f423cc8b198
generation 2 starts at 7f43c7fff000
ephemeral segment allocation context: none
Small object heap
         segment            begin        allocated        committed allocated size         committed size        
    7f419d080000     7f419d081000     7f41acef9e78     7f41acf1a000 0xfe78e78 (266833528)  0xfe9a000 (266969088) 
    7f41bbfe0000     7f41bbfe1000     7f41cbb9b950     7f41cbbbc000 0xfbba950 (263956816)  0xfbdc000 (264093696) 
    7f4203ff0000     7f4203ff1000     7f4213feffb0     7f4213ff0000 0xfffefb0 (268431280)  0x10000000 (268435456)
    7f422bffa000     7f422bffb000     7f423a3a02f0     7f423a3c1000 0xe3a52f0 (238703344)  0xe3c7000 (238841856) 
    7f428bff0000     7f428bff1000     7f429bfeffe0     7f429bff0000 0xfffefe0 (268431328)  0x10000000 (268435456)
    7f429bff2000     7f429bff3000     7f42abff2000     7f42abff2000 0xffff000 (268431360)  0x10000000 (268435456)
    7f42abffe000     7f42abfff000     7f42bbffdfd0     7f42bbffe000 0xfffefd0 (268431312)  0x10000000 (268435456)
    7f42d3ffe000     7f42d3fff000     7f42e3ffe000     7f42e3ffe000 0xffff000 (268431360)  0x10000000 (268435456)
    7f42fb030000     7f42fb031000     7f430b030000     7f430b030000 0xffff000 (268431360)  0x10000000 (268435456)
    7f4317ffe000     7f4317fff000     7f4327ffe000     7f4327ffe000 0xffff000 (268431360)  0x10000000 (268435456)
    7f43c7ffe000     7f43c7fff000     7f43d7ffdfe8     7f43d7ffe000 0xfffefe8 (268431336)  0x10000000 (268435456)
    7f423bffe000     7f423bfff000     7f423d33f780     7f423e51b000 0x1340780 (20187008)   0x251d000 (38916096)  
Large object heap starts at 7f43d7fff000
         segment            begin        allocated        committed allocated size         committed size        
    7f40ed054000     7f40ed055000     7f40f1055038     7f40f1056000 0x4000038 (67108920)   0x4002000 (67117056)  
    7f41d3fe4000     7f41d3fe5000     7f41d7fe5038     7f41d8006000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41dbfe6000     7f41dbfe7000     7f41dffe7038     7f41e0008000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41ebfea000     7f41ebfeb000     7f41effeb038     7f41f000c000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41f3fec000     7f41f3fed000     7f41f7fed038     7f41f800e000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41fbfee000     7f41fbfef000     7f41fffef038     7f4200010000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f432fffe000     7f432ffff000     7f4337063918     7f4337084000 0x7064918 (117852440)  0x7086000 (117989376) 
    7f43d7ffe000     7f43d7fff000     7f43de5eb030     7f43de5ec000 0x65ec030 (106872880)  0x65ee000 (106881024) 
Pinned object heap starts at 7f43dffff000
         segment            begin        allocated        committed allocated size         committed size        
    7f43dfffe000     7f43dffff000     7f43e02bc4f8     7f43e02c0000 0x2bd4f8 (2872568)     0x2c2000 (2891776)    
------------------------------
GC Allocated Heap Size:    Size: 0xd4a1f110 (3567382800) bytes.
GC Committed Heap Size:    Size: 0xd5d3c000 (3587424256) bytes.

Total bytes consumed by CLR: 0xd827a000 (3626475520)

可以看到,CLR 吃掉了3.6G。Ok,到这儿基本可以确定是托管内存问题。

相信大家都注意到小对象堆和大对象堆有大量的内存段,吃掉了主要的托管内存。

接下来使用 dumpheap -stat 来统计下托管堆上内存所有对象:

0:000> !sos dumpheap -stat
Statistics:
          MT      Count   TotalSize Class Name
7f4406bf5ef0          1          24 NPOI.OpenXmlFormats.ExtendedPropertiesDocument
7f4406bf60a0          1          24 NPOI.OpenXmlFormats.CustomPropertiesDocument
7f4406bf67e8          1          24 NPOI.OpenXmlFormats.CT_CustomProperties
7f4406bf74e0          1          24 NPOI.OpenXmlFormats.CT_Property[]
7f4406bf8540          1          24 NPOI.HSSF.Util.HSSFColor+Black
7f4406bf8678          1          24 NPOI.HSSF.Util.HSSFColor+White
...
...
7f440213aa08      3,270     313,920 System.Reflection.Emit.DynamicMethod
7f44000c8080      4,135     358,828 System.Int32[]
7f4401519498     10,309     365,192 System.RuntimeType[]
7f440480b990          2     458,800 StackExchange.Redis.RawResult[]
7f440213faa0      3,270     470,880 System.Reflection.Emit.DynamicILGenerator
7f440185dcf8      5,133     492,768 System.Reflection.RuntimeParameterInfo
7f44000e7e60      3,216     551,160 System.String[]
7f440185e0a8      6,986     558,880 System.Signature
7f44012e8f10     10,812   1,124,448 System.Reflection.RuntimeMethodInfo
7f44000c7bc8     15,112   1,305,648 System.SByte[]
7f44000cd2e0     30,854   2,294,568 System.String
7f4400110eb0        224   5,064,502 System.Char[]
7f4400b784f0     18,748  21,734,523 System.Byte[]
7f44046efe70  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.EndPointEventArgs>
7f44047001b0  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.HashSlotMovedEventArgs>
7f4404700488  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.InternalErrorEventArgs>
7f44046efc10  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.RedisErrorEventArgs>
5558fbe656a0    293,562 474,504,032 Free
7f440001b0f8      5,222 538,033,592 System.Object[]
7f44046ef988 13,082,904 837,305,856 System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs>
Total 39,789,084 objects, 3,567,106,882 bytes

这个命令会按照 mt 汇总统计每个对象的数量及所占内存大小,并按内存大小排序,我们看到询问有几个对象格外抢眼,其中最后一个有 1000w 个 EvevtHandler 对象,占 837 M,那就继续往下看。

罪魁祸首

我们需要看一下这 1000w 对象到底什么来头,因为上述是按 mt 汇总的,所以现在用 dumpheap -mt 展开看一下:

0:000> !sos dumpheap -mt 7f44046ef988
         Address               MT           Size
    7f419d0810c0     7f44046ef988             64 
    7f419d081100     7f44046ef988             64 
    7f419d081240     7f44046ef988             64  
...
    7f419d2f0e50     7f44046ef988             64 
    7f419d2f0f90     7f44046ef988             64 
    7f419d2f0fd0     7f44046ef988             64 

Statistics:
          MT  Count TotalSize Class Name
7f44046ef988 13,160   842,240 System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs>
Total 13,160 objects, 842,240 bytes

对象太多,不可能展示完,所以这里做了一个省略。好的,接下来抽查几个对象使用命令 dumpobj 看一下:

0:000> !sos dumpobj /d 7f419d2f0fd0
Name:        System.EventHandler`1[[StackExchange.Redis.ConnectionFailedEventArgs, StackExchange.Redis]]
MethodTable: 00007f44046ef988
EEClass:     00007f44046f8b58
Tracked Type: false
Size:        64(0x40) bytes
File:        /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.6/System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f4400015290  40001bd        8        System.Object  0 instance 00007f419d2f0fd0 _target
00007f4400015290  40001be       10        System.Object  0 instance 0000000000000000 _methodBase
00007f44000c3eb8  40001bf       18        System.IntPtr  1 instance 00007F43FF4ED1E0 _methodPtr
00007f44000c3eb8  40001c0       20        System.IntPtr  1 instance 00007F4403BEF508 _methodPtrAux
00007f4400015290  4000243       28        System.Object  0 instance 0000000000000000 _invocationList
00007f44000c3eb8  4000244       30        System.IntPtr  1 instance 0000000000000000 _invocationCount

抽查几个后发现,都是一样。那既然这些对象一直不被GC回收,肯定有什么原因,使用 gcroot 查看一下都被谁引用:

0:000> !sos gcroot 7f419d2f0fd0
Caching GC roots, this may take a while.
Subsequent runs of this command will be faster.

HandleTable:
    00007f447a911310 (strong handle)
          -> 7f43e0008c28     System.Object[] 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer (static variable: System.Object._locker)
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

Thread 1f4f:
    7f44724de9b0 7f440488ac93 StackExchange.Redis.PhysicalBridge+<ProcessBacklogAsync>d__97.MoveNext()
        rbp-78: 00007f44724de9c8
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

        rbp-80: 00007f44724de9c0
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724dea50 7f44048888f9
        rbp-8: 00007f44724dea78
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724dea90 7f44048882fb
        rbp-10: 00007f44724dea90
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724deab0 7f44048882ac StackExchange.Redis.PhysicalBridge.ProcessBacklogAsync()
        rbp-30: 00007f44724deab0
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

        rbp-8: 00007f44724dead8
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724deaf0 7f4404887f3f StackExchange.Redis.PhysicalBridge+<>c.<StartBacklogProcessor>b__93_0(System.Object)
        rbp-10: 00007f44724deaf0
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

Found 8 unique roots.

注意:因为对象太多,遍历查找引用链需要一点儿时间,请耐心等待!

我们发现,它们都被 StackExchange.Redis.ConnectionMultiplexer 引用,这是一个 redis 连接对象。再经过大胆猜测和简单分析,其余几个巨量对象列表都是这个 redis 对象的事件。说明程序中引入了 StackExchange 的 redis 扩展包,并在项目中某处使用了 redis 操作,这个对象到底在何处,可能出问题的代码又在哪儿?

定位代码

有经验的朋友可能会知道,redis 这个对象大概率是一个静态变量。现在我们来验证一下,试图查找这个对象对应的静态变量所在位置,使用 s-q 命令查找是哪个内存地址(B)上保存此这个 redis 对象地址(A)。

首先缩小内存查找范围:

0:000> !sos dumpobj /d 7f43e0008c28
Name:        System.Object[]
MethodTable: 00007f440001b0f8
EEClass:     00007f440001b070
Tracked Type: false
Size:        32664(0x7f98) bytes
Array:       Rank 1, Number of elements 4080, Type CLASS (Print Array)
Fields:
None

我们需要用到这个 Object 对象数组的起始位置及数量用作如下命令的偏移:

0:000> s-q 7f43e0008c28 L?0x7f98 7f43c81a4aa0
00007f43`e000aaa0  00007f43`c81a4aa0 00007f43`ca4922a0

这样,我们就找到了静态变量的位置(B):

00007f43`e000aaa0

让我们继续查找是哪些代码在使用这个静态变量:

0:000> s-b 0 L?0xffffffffffffffff a0 aa 00 e0 43 7f 00 00
00007f44`044f9f81  a0 aa 00 e0 43 7f 00 00-33 ff 48 89 38 48 b8 6c  ....C...3.H.8H.l
00007f44`044fb96a  a0 aa 00 e0 43 7f 00 00-48 83 3e 00 40 0f 94 c6  ....C...H.>.@...
00007f44`044fb9b3  a0 aa 00 e0 43 7f 00 00-48 83 3f 00 74 38 48 bf  ....C...H.?.t8H.
00007f44`044fb9c3  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 b8 28 b0 6e  ....C...H.?H.(.n
00007f44`044fba71  a0 aa 00 e0 43 7f 00 00-48 8b b5 50 ff ff ff e8  ....C...H..P....
00007f44`044fbaa9  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d b8 48  ....C...H.?H.}.H
00007f44`044fbb03  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d a8 48  ....C...H.?H.}.H
00007f44`044fbb5d  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d 98 48  ....C...H.?H.}.H
00007f44`044fbbb7  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d 88 48  ....C...H.?H.}.H
00007f44`044fbc11  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 bd 78 ff  ....C...H.?H..x.
00007f44`044fbc74  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 bd 68 ff  ....C...H.?H..h.
00007f44`044fbcc7  a0 aa 00 e0 43 7f 00 00-48 8b 00 48 89 45 d0 90  ....C...H..H.E..

OK 我们找到不少,抽几个看一下反汇编代码:

0:000> !sos u 00007f44`044fb96a
Normal JIT generated code
Wisder.xRiver.BaseCore.Util.xRedis.get_Instance()
ilAddr is 00007F447142F990 pImport is 00000143049FEE00
Begin 00007F44044FB890, size 485
00007f44`044fb890 55              push    rbp
00007f44`044fb891 4881ecd0000000  sub     rsp,0D0h
00007f44`044fb898 488dac24d0000000 lea     rbp,[rsp+0D0h]
00007f44`044fb8a0 33c0            xor     eax,eax
00007f44`044fb8a2 48898538ffffff  mov     qword ptr [rbp-0C8h],rax
00007f44`044fb8a9 c4413857c0      vxorps  xmm8,xmm8,xmm8
00007f44`044fb8ae 48b840ffffffffffffff mov rax,0FFFFFFFFFFFFFF40h
00007f44`044fb8b8 c5797f0428      vmovdqa xmmword ptr [rax+rbp],xmm8
00007f44`044fb8bd c5797f440510    vmovdqa xmmword ptr [rbp+rax+10h],xmm8
00007f44`044fb8c3 c5797f440520    vmovdqa xmmword ptr [rbp+rax+20h],xmm8
00007f44`044fb8c9 4883c030        add     rax,30h
00007f44`044fb8cd 75e9            jne     00007f44`044fb8b8
00007f44`044fb8cf 4889a530ffffff  mov     qword ptr [rbp-0D0h],rsp
00007f44`044fb8d6 48b870e26402447f0000 mov rax,7F440264E270h
00007f44`044fb8e0 833800          cmp     dword ptr [rax],0
00007f44`044fb8e3 7405            je      00007f44`044fb8ea
00007f44`044fb8e5 e88659dd74      call    libcoreclr!JIT_DbgIsJustMyCode (00007f44`792d1270)
00007f44`044fb8ea 90              nop
00007f44`044fb8eb 48bf90aa00e0437f0000 mov rdi,7F43E000AA90h
00007f44`044fb8f5 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fb8f8 393f            cmp     dword ptr [rdi],edi
00007f44`044fb8fa e829ccb2fb      call    00007f44`00028528 (System.String.get_Length(), mdToken: 0000000006000764)
00007f44`044fb8ff 8945cc          mov     dword ptr [rbp-34h],eax
00007f44`044fb902 837dcc00        cmp     dword ptr [rbp-34h],0
00007f44`044fb906 400f94c6        sete    sil
00007f44`044fb90a 400fb6f6        movzx   esi,sil
00007f44`044fb90e 8975fc          mov     dword ptr [rbp-4],esi
00007f44`044fb911 837dfc00        cmp     dword ptr [rbp-4],0
00007f44`044fb915 7451            je      00007f44`044fb968
00007f44`044fb917 90              nop
00007f44`044fb918 48bf90ec0c00447f0000 mov rdi,7F44000CEC90h (MT: System.Exception)
00007f44`044fb922 e809c8dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fb927 48898540ffffff  mov     qword ptr [rbp-0C0h],rax
00007f44`044fb92e bf49220000      mov     edi,2249h
00007f44`044fb933 48be00de6402447f0000 mov rsi,7F440264DE00h
00007f44`044fb93d e87ecedc74      call    libcoreclr!JIT_StrCns (00007f44`792c87c0)
00007f44`044fb942 48898538ffffff  mov     qword ptr [rbp-0C8h],rax
00007f44`044fb949 488bb538ffffff  mov     rsi,qword ptr [rbp-0C8h]
00007f44`044fb950 488bbd40ffffff  mov     rdi,qword ptr [rbp-0C0h]
00007f44`044fb957 e89cd1b2fb      call    00007f44`00028af8 (System.Exception..ctor(System.String), mdToken: 00000000060003CB)
00007f44`044fb95c 488bbd40ffffff  mov     rdi,qword ptr [rbp-0C0h]
00007f44`044fb963 e8f832dd74      call    libcoreclr!IL_Throw (00007f44`792cec60)
00007f44`044fb968 48bea0aa00e0437f0000 mov rsi,7F43E000AAA0h
00007f44`044fb972 48833e00        cmp     qword ptr [rsi],0
00007f44`044fb976 400f94c6        sete    sil
00007f44`044fb97a 400fb6f6        movzx   esi,sil
00007f44`044fb97e 8975f8          mov     dword ptr [rbp-8],esi
00007f44`044fb981 837df800        cmp     dword ptr [rbp-8],0
00007f44`044fb985 0f8409010000    je      00007f44`044fba94
00007f44`044fb98b 90              nop
00007f44`044fb98c 48be98aa00e0437f0000 mov rsi,7F43E000AA98h
00007f44`044fb996 488b36          mov     rsi,qword ptr [rsi]
00007f44`044fb999 488975f0        mov     qword ptr [rbp-10h],rsi
00007f44`044fb99d 33f6            xor     esi,esi
00007f44`044fb99f 8975e8          mov     dword ptr [rbp-18h],esi
00007f44`044fb9a2 488d75e8        lea     rsi,[rbp-18h]
00007f44`044fb9a6 488b7df0        mov     rdi,qword ptr [rbp-10h]
00007f44`044fb9aa e811dbb2fb      call    00007f44`000294c0 (System.Threading.Monitor.Enter(System.Object, Boolean ByRef), mdToken: 0000000006002BA0)
00007f44`044fb9af 90              nop
00007f44`044fb9b0 90              nop
00007f44`044fb9b1 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fb9bb 48833f00        cmp     qword ptr [rdi],0
00007f44`044fb9bf 7438            je      00007f44`044fb9f9
00007f44`044fb9c1 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fb9cb 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fb9ce 48b828b06e04447f0000 mov rax,7F44046EB028h
00007f44`044fb9d8 393f            cmp     dword ptr [rdi],edi
00007f44`044fb9da ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.get_IsConnected(), mdToken: 00000000060001A2)
00007f44`044fb9dc 89854cffffff    mov     dword ptr [rbp-0B4h],eax
00007f44`044fb9e2 83bd4cffffff00  cmp     dword ptr [rbp-0B4h],0
00007f44`044fb9e9 400f94c7        sete    dil
00007f44`044fb9ed 400fb6ff        movzx   edi,dil
00007f44`044fb9f1 89bd64ffffff    mov     dword ptr [rbp-9Ch],edi
00007f44`044fb9f7 eb0a            jmp     00007f44`044fba03
00007f44`044fb9f9 c78564ffffff01000000 mov dword ptr [rbp-9Ch],1
00007f44`044fba03 8bbd64ffffff    mov     edi,dword ptr [rbp-9Ch]
00007f44`044fba09 400fb6ff        movzx   edi,dil
00007f44`044fba0d 897de4          mov     dword ptr [rbp-1Ch],edi
00007f44`044fba10 837de400        cmp     dword ptr [rbp-1Ch],0
00007f44`044fba14 7470            je      00007f44`044fba86
00007f44`044fba16 90              nop
00007f44`044fba17 48bf90aa00e0437f0000 mov rdi,7F43E000AA90h
00007f44`044fba21 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fba24 e86ff6ffff      call    00007f44`044fb098 (StackExchange.Redis.ConfigurationOptions.Parse(System.String), mdToken: 0000000006000156)
00007f44`044fba29 48898558ffffff  mov     qword ptr [rbp-0A8h],rax
00007f44`044fba30 488bbd58ffffff  mov     rdi,qword ptr [rbp-0A8h]
00007f44`044fba37 48897dd8        mov     qword ptr [rbp-28h],rdi
00007f44`044fba3b 488b7dd8        mov     rdi,qword ptr [rbp-28h]
00007f44`044fba3f be10270000      mov     esi,2710h
00007f44`044fba44 393f            cmp     dword ptr [rdi],edi
00007f44`044fba46 e8edf7ffff      call    00007f44`044fb238 (StackExchange.Redis.ConfigurationOptions.set_ConnectTimeout(Int32), mdToken: 0000000006000121)
00007f44`044fba4b 90              nop
00007f44`044fba4c 488b7dd8        mov     rdi,qword ptr [rbp-28h]
00007f44`044fba50 be983a0000      mov     esi,3A98h
00007f44`044fba55 393f            cmp     dword ptr [rdi],edi
00007f44`044fba57 e82cf9ffff      call    00007f44`044fb388 (StackExchange.Redis.ConfigurationOptions.set_SyncTimeout(Int32), mdToken: 000000000600014B)
00007f44`044fba5c 90              nop
00007f44`044fba5d 488b7dd8        mov     rdi,qword ptr [rbp-28h]
00007f44`044fba61 33f6            xor     esi,esi
00007f44`044fba63 e838e3ffff      call    00007f44`044f9da0 (StackExchange.Redis.ConnectionMultiplexer.Connect(StackExchange.Redis.ConfigurationOptions, System.IO.TextWriter), mdToken: 00000000060001BC)
00007f44`044fba68 48898550ffffff  mov     qword ptr [rbp-0B0h],rax
00007f44`044fba6f 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fba79 488bb550ffffff  mov     rsi,qword ptr [rbp-0B0h]
00007f44`044fba80 e80cb7f274      call    libcoreclr!JIT_CheckedWriteBarrier (00007f44`79427191)
00007f44`044fba85 90              nop
00007f44`044fba86 90              nop
00007f44`044fba87 90              nop
00007f44`044fba88 eb00            jmp     00007f44`044fba8a
00007f44`044fba8a 488bfc          mov     rdi,rsp
00007f44`044fba8d e854020000      call    00007f44`044fbce6 (Wisder.xRiver.BaseCore.Util.xRedis.get_Instance(), mdToken: 00000000060000B3)
00007f44`044fba92 90              nop
00007f44`044fba93 90              nop
00007f44`044fba94 48bf88f96e04447f0000 mov rdi,7F44046EF988h (MT: System.EventHandler`1[[StackExchange.Redis.ConnectionFailedEventArgs, StackExchange.Redis]])
00007f44`044fba9e e88dc6dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbaa3 488945c0        mov     qword ptr [rbp-40h],rax
00007f44`044fbaa7 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbab1 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbab4 48897db8        mov     qword ptr [rbp-48h],rdi
00007f44`044fbab8 488b7dc0        mov     rdi,qword ptr [rbp-40h]
00007f44`044fbabc 33f6            xor     esi,esi
00007f44`044fbabe 48ba10f5be03447f0000 mov rdx,7F4403BEF510h
00007f44`044fbac8 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbad2 e8b1b6b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbad7 488b7db8        mov     rdi,qword ptr [rbp-48h]
00007f44`044fbadb 488b75c0        mov     rsi,qword ptr [rbp-40h]
00007f44`044fbadf 48b818b16e04447f0000 mov rax,7F44046EB118h
00007f44`044fbae9 393f            cmp     dword ptr [rdi],edi
00007f44`044fbaeb ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ConnectionFailed(System.EventHandler`1<StackExchange.Redis.ConnectionFailedEventArgs>), mdToken: 00000000060001F9)
00007f44`044fbaed 90              nop
00007f44`044fbaee 48bf88f96e04447f0000 mov rdi,7F44046EF988h (MT: System.EventHandler`1[[StackExchange.Redis.ConnectionFailedEventArgs, StackExchange.Redis]])
00007f44`044fbaf8 e833c6dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbafd 488945b0        mov     qword ptr [rbp-50h],rax
00007f44`044fbb01 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbb0b 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbb0e 48897da8        mov     qword ptr [rbp-58h],rdi
00007f44`044fbb12 488b7db0        mov     rdi,qword ptr [rbp-50h]
00007f44`044fbb16 33f6            xor     esi,esi
00007f44`044fbb18 48ba08f5be03447f0000 mov rdx,7F4403BEF508h
00007f44`044fbb22 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbb2c e857b6b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbb31 488b7da8        mov     rdi,qword ptr [rbp-58h]
00007f44`044fbb35 488b75b0        mov     rsi,qword ptr [rbp-50h]
00007f44`044fbb39 48b838b16e04447f0000 mov rax,7F44046EB138h
00007f44`044fbb43 393f            cmp     dword ptr [rdi],edi
00007f44`044fbb45 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ConnectionRestored(System.EventHandler`1<StackExchange.Redis.ConnectionFailedEventArgs>), mdToken: 00000000060001FF)
00007f44`044fbb47 90              nop
00007f44`044fbb48 48bf10fc6e04447f0000 mov rdi,7F44046EFC10h (MT: System.EventHandler`1[[StackExchange.Redis.RedisErrorEventArgs, StackExchange.Redis]])
00007f44`044fbb52 e8d9c5dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbb57 488945a0        mov     qword ptr [rbp-60h],rax
00007f44`044fbb5b 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbb65 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbb68 48897d98        mov     qword ptr [rbp-68h],rdi
00007f44`044fbb6c 488b7da0        mov     rdi,qword ptr [rbp-60h]
00007f44`044fbb70 33f6            xor     esi,esi
00007f44`044fbb72 48ba00f5be03447f0000 mov rdx,7F4403BEF500h
00007f44`044fbb7c 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbb86 e8fdb5b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbb8b 488b7d98        mov     rdi,qword ptr [rbp-68h]
00007f44`044fbb8f 488b75a0        mov     rsi,qword ptr [rbp-60h]
00007f44`044fbb93 48b878b16e04447f0000 mov rax,7F44046EB178h
00007f44`044fbb9d 393f            cmp     dword ptr [rdi],edi
00007f44`044fbb9f ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ErrorMessage(System.EventHandler`1<StackExchange.Redis.RedisErrorEventArgs>), mdToken: 000000000600020F)
00007f44`044fbba1 90              nop
00007f44`044fbba2 48bf70fe6e04447f0000 mov rdi,7F44046EFE70h (MT: System.EventHandler`1[[StackExchange.Redis.EndPointEventArgs, StackExchange.Redis]])
00007f44`044fbbac e87fc5dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbbb1 48894590        mov     qword ptr [rbp-70h],rax
00007f44`044fbbb5 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbbbf 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbbc2 48897d88        mov     qword ptr [rbp-78h],rdi
00007f44`044fbbc6 488b7d90        mov     rdi,qword ptr [rbp-70h]
00007f44`044fbbca 33f6            xor     esi,esi
00007f44`044fbbcc 48baf8f4be03447f0000 mov rdx,7F4403BEF4F8h
00007f44`044fbbd6 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbbe0 e8a3b5b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbbe5 488b7d88        mov     rdi,qword ptr [rbp-78h]
00007f44`044fbbe9 488b7590        mov     rsi,qword ptr [rbp-70h]
00007f44`044fbbed 48b848b16e04447f0000 mov rax,7F44046EB148h
00007f44`044fbbf7 393f            cmp     dword ptr [rdi],edi
00007f44`044fbbf9 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ConfigurationChanged(System.EventHandler`1<StackExchange.Redis.EndPointEventArgs>), mdToken: 0000000006000202)
00007f44`044fbbfb 90              nop
00007f44`044fbbfc 48bfb0017004447f0000 mov rdi,7F44047001B0h (MT: System.EventHandler`1[[StackExchange.Redis.HashSlotMovedEventArgs, StackExchange.Redis]])
00007f44`044fbc06 e825c5dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbc0b 48894580        mov     qword ptr [rbp-80h],rax
00007f44`044fbc0f 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbc19 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbc1c 4889bd78ffffff  mov     qword ptr [rbp-88h],rdi
00007f44`044fbc23 488b7d80        mov     rdi,qword ptr [rbp-80h]
00007f44`044fbc27 33f6            xor     esi,esi
00007f44`044fbc29 48ba18f5be03447f0000 mov rdx,7F4403BEF518h
00007f44`044fbc33 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbc3d e846b5b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbc42 488bbd78ffffff  mov     rdi,qword ptr [rbp-88h]
00007f44`044fbc49 488b7580        mov     rsi,qword ptr [rbp-80h]
00007f44`044fbc4d 48b868b16e04447f0000 mov rax,7F44046EB168h
00007f44`044fbc57 393f            cmp     dword ptr [rdi],edi
00007f44`044fbc59 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_HashSlotMoved(System.EventHandler`1<StackExchange.Redis.HashSlotMovedEventArgs>), mdToken: 000000000600020C)
00007f44`044fbc5b 90              nop
00007f44`044fbc5c 48bf88047004447f0000 mov rdi,7F4404700488h (MT: System.EventHandler`1[[StackExchange.Redis.InternalErrorEventArgs, StackExchange.Redis]])
00007f44`044fbc66 e8c5c4dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbc6b 48898570ffffff  mov     qword ptr [rbp-90h],rax
00007f44`044fbc72 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbc7c 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbc7f 4889bd68ffffff  mov     qword ptr [rbp-98h],rdi
00007f44`044fbc86 488bbd70ffffff  mov     rdi,qword ptr [rbp-90h]
00007f44`044fbc8d 33f6            xor     esi,esi
00007f44`044fbc8f 48ba20f5be03447f0000 mov rdx,7F4403BEF520h
00007f44`044fbc99 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbca3 e8e0b4b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbca8 488bbd68ffffff  mov     rdi,qword ptr [rbp-98h]
00007f44`044fbcaf 488bb570ffffff  mov     rsi,qword ptr [rbp-90h]
00007f44`044fbcb6 48b828b16e04447f0000 mov rax,7F44046EB128h
00007f44`044fbcc0 393f            cmp     dword ptr [rdi],edi
00007f44`044fbcc2 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_InternalError(System.EventHandler`1<StackExchange.Redis.InternalErrorEventArgs>), mdToken: 00000000060001FC)
00007f44`044fbcc4 90              nop
00007f44`044fbcc5 48b8a0aa00e0437f0000 mov rax,7F43E000AAA0h
00007f44`044fbccf 488b00          mov     rax,qword ptr [rax]
00007f44`044fbcd2 488945d0        mov     qword ptr [rbp-30h],rax
00007f44`044fbcd6 90              nop
00007f44`044fbcd7 eb00            jmp     00007f44`044fbcd9
00007f44`044fbcd9 488b45d0        mov     rax,qword ptr [rbp-30h]
00007f44`044fbcdd 4881c4d0000000  add     rsp,0D0h
00007f44`044fbce4 5d              pop     rbp
00007f44`044fbce5 c3              ret
00007f44`044fbce6 55              push    rbp
00007f44`044fbce7 4883ec10        sub     rsp,10h
00007f44`044fbceb 488b2f          mov     rbp,qword ptr [rdi]
00007f44`044fbcee 48892c24        mov     qword ptr [rsp],rbp
00007f44`044fbcf2 488dadd0000000  lea     rbp,[rbp+0D0h]
00007f44`044fbcf9 8b7de8          mov     edi,dword ptr [rbp-18h]
00007f44`044fbcfc 400fb6ff        movzx   edi,dil
00007f44`044fbd00 85ff            test    edi,edi
00007f44`044fbd02 740a            je      00007f44`044fbd0e
00007f44`044fbd04 488b7df0        mov     rdi,qword ptr [rbp-10h]
00007f44`044fbd08 e83317dd74      call    libcoreclr!JIT_MonExit_Portable (00007f44`792cd440)
00007f44`044fbd0d 90              nop
00007f44`044fbd0e 90              nop
00007f44`044fbd0f 4883c410        add     rsp,10h
00007f44`044fbd13 5d              pop     rbp
00007f44`044fbd14 c3              ret

重点来了,Wisder.xRiver.BaseCore.Util.xRedis.get_Instance() 这个方法其实是 Instance 属性的 get 方法。

可以继续使用 ip2md 命令将指令地址转换出模块信息:

0:000> !sos ip2md 00007f44`044fb96a
MethodDesc:   00007f4403caf880
Method Name:          Wisder.xRiver.BaseCore.Util.xRedis.get_Instance()
Class:                00007f4403d1a278
MethodTable:          00007f4403cafe48
mdToken:              00000000060000B3
Module:               00007f440264de00
IsJitted:             yes
Current CodeAddr:     00007f44044fb890
Version History:
  ILCodeVersion:      0000000000000000
  ReJIT ID:           0
  IL Addr:            00007f447142f990
     CodeAddr:           00007f44044fb890  (MinOptJitted)
     NativeCodeVersion:  0000000000000000

找到模块后,查看一下信息就可以找到对应的 dll 文件:

0:000> !sos DumpModule /d 00007f440264de00
Name: /xxx/xRiver.BaseCore.dll
Attributes:              PEFile IsFileLayout 
TransientFlags:          00209011 
Assembly:                00005558fc05f540
BaseAddress:             00007F447142A000
PEAssembly:              00005558FC03ABE0
ModuleId:                00007F440264E990
ModuleIndex:             0000000000000064
LoaderHeap:              0000000000000000
TypeDefToMethodTableMap: 00007F4402650000
TypeRefToMethodTableMap: 00007F4402650560
MethodDefToDescMap:      00007F4402650EE8
FieldDefToDescMap:       00007F4402653550
MemberRefToDescMap:      0000000000000000
FileReferencesMap:       00007F4402655948
AssemblyReferencesMap:   00007F4402655950
MetaData start address:  00007F4471444540 (148124 bytes)

那么接下来就是找到这个 dll 文件对应的源码项目,重点关注一下里面 get_Instance 方法。

namespace Wisder.xRiver.BaseCore.Util
{
    public static class xRedis
    {
        private static ConnectionMultiplexer _instance = null;

        /// <summary>
        /// 使用一个静态属性来返回已连接的实例,如下列中所示。
        /// 这样,一旦 ConnectionMultiplexer 断开连接,便可以初始化新的连接实例。
        /// </summary>
        public static ConnectionMultiplexer Instance
        {
            get
            {
                if (Constr.Length == 0)
                {
                    throw new Exception("Redis连接字符串未设置!");
                }

                if (_instance == null)
                {
                    lock (_locker)
                    {
                        if (_instance == null || !_instance.IsConnected)
                        {
                            var options = ConfigurationOptions.Parse(Constr);
                            options.ConnectTimeout = 10000; // 设置连接超时时间为10秒
                            options.SyncTimeout = 15000; // 设置同步操作的超时时间为15秒

                            _instance = ConnectionMultiplexer.Connect(options); //Constr
                        }
                    }
                }

                //注册如下事件
                _instance.ConnectionFailed += MuxerConnectionFailed;
                _instance.ConnectionRestored += MuxerConnectionRestored;
                _instance.ErrorMessage += MuxerErrorMessage;
                _instance.ConfigurationChanged += MuxerConfigurationChanged;
                _instance.HashSlotMoved += MuxerHashSlotMoved;
                _instance.InternalError += MuxerInternalError;
                return _instance;
            }
        }
    }
}

解决思路:把事件注册挪到最里面的 if 内部,_instance 赋值的后面。后续修改后可以观察下情况。

总结分析

一个事件注册会产生一个 64 Byte 大小的对象,不要小看它,积少成多,照样会导致内存泄露问题。

这让我不由得想起了 PHP 语言,一个请求结束就会销毁进程,可有效避免内存泄露问题,实在不行再重启一下 php 服务,也可以解决内存泄露问题。回到 .net 这一类平台上,就可以好好写代码,提高代码质量,避免内存泄露等问题。

posted @ 2024-07-18 10:44  successgo  阅读(1183)  评论(10编辑  收藏  举报